API docs: fresco.util

fresco.util.common

Common utilities for writing applications in fresco

fresco.util.common.object_or_404(ob, exception=<class 'fresco.exceptions.NotFound'>)

Return the value of ob if it is not None. Otherwise raise a NotFound exception.

fresco.util.security

Security-related utilities

fresco.util.security.check_equal_constant_time(a, b)

Return True if string a is equal to string b.

If a and b are of the same length, this function will take the same amount of time to execute, regardless of whether or not a and b are equal.

fresco.util.http

Utilities for working with data on the HTTP level

fresco.util.http.ALLOWED_ENCODINGS = {'latin7', '932', 'cp1256', 'iso-8859-10', 'sjis2004', 'mskanji', 'maccyrillic', 'ibm037', 'iso-8859-8', 'windows-1254', '860', 'latin6', 'jisx0213', '273', 'macgreek', 'cp850', 'latin-1', 'hebrew', 'iso8859-5', 'cp866', 'mac-turkish', 'cp857', 'cp950', 'iso-2022-jp-2', 'macturkish', 'cp154', 'hkscs', 's-jisx0213', 'cp860', 'iso-2022-jp-3', 'cp949', 'iso8859-4', 'iso-8859-3', 'cp1251', 'cp1257', 'l5', 'utf-32le', 'gb18030', 'cp1125', 'ibm861', 'koi8-r', 'cp864', 'cp861', '857', 'gb2312-1980', 'macroman', 'ibm858', 'urdu', 'ibm864', 'utf-7', 'euc-kr', 'iso-8859-16', '865', 'cp862', '8859', 'cyrillic-asian', 'iso2022jp-2004', 'gb18030-2000', 'utf8', 'ibm852', 'ibm850', 'cp273', 'iso8859-13', 'iso8859-14', 'ascii', 'iso2022-jp-3', 'maciceland', '863', '646', 'ibm273', 'maclatin2', 'csbig5', 'l2', 'euc-jisx0213', 'ibm866', 'utf', 'windows-1253', 'latin', 'ibm500', 'u7', 'ibm862', 'csibm273', 'utf-32', 'korean', 'iso-2022-kr', 'cp775', '858', 'ibm865', 'ibm775', 'ms932', 'hz', 'greek8', 'iso2022jp-3', 'iso8859-2', '950', 'ibm1140', 'csiso2022jp', 'latin2', 'iso-ir-58', 'iso-8859-14', 'ibm860', 'iso8859-9', 'l6', '852', 'cp1254', 'l9', 'u8', 'iso2022-jp-ext', 'ks-x-1001', 'iso2022kr', 'mac-cyrillic', 'russian', 'arabic', '869', 'big5-tw', 'latin5', 'cp-is', 'iso8859-1', 'euc-jp', 'iso2022-jp-2', 'iso-8859-7', 'maccentraleurope', 'utf-8', 'iso2022-jp-2004', 'iso8859-8', 'latin3', 'euc-cn', 'windows-1256', 'windows-1258', 'hzgb', 'cp852', 'windows-1250', 'cp819', 'ibm857', 'shiftjis', 'ujis', 'ibm855', 'eucjisx0213', 'utf-16be', 'cp863', 'latin8', 'l1', '437', 'cp500', 'iso-8859-5', 'ibm863', 'iso8859-15', 'thai', 'u16', 'iso2022jp-ext', 'cp1258', 'ibm424', 'ptcp154', 'euc-jis-2004', 'utf-16-le', 'csshiftjis', 'cp720', '866', 'shiftjisx0213', 'ms-kanji', 'cp1140', 'mac-latin2', 'sjis', 'big5-hkscs', 'cp037', 'iso-8859-4', 'ibm039', 'euccn', 'utf-32-be', 'gbk', 'latin1', 'iso-8859-13', 'ms1361', 'iso2022-jp', 'l4', 'big5', 'ibm1125', 'windows-1257', 'shift-jis', 'eucgb2312-cn', 'hz-gb-2312', 'cp855', 'iso2022jp', 'iso-8859-15', 'koi8-u', 'u32', 'ks-c-5601-1987', 'iso2022-kr', 'latin4', 'cp1361', 'iso2022jp-1', 'utf-16le', 'windows-1251', 's-jis', 'utf16', 'ms950', 'mac-roman', 'chinese', 'l10', 'cp858', 'cp856', 'u-jis', 'csiso2022kr', 'pt154', 'ibm437', 'windows-1252', 'mac-iceland', 'cp1253', 'iso-2022-jp', 'cp932', 'gb2312-80', 'cp424', '855', 'ksx1001', 'sjis-2004', 'sjisx0213', 'iso-8859-1', 'cp1252', 'latin9', 'iso-8859-9', 'iso-2022-jp-ext', 'utf-32be', 'iso8859-3', 'ms936', 'euckr', 'ukrainian', '862', 'gb2312', 'ibm1026', 'cp936', 'l8', 'us-ascii', 'eucjp', 'l3', 'shiftjis2004', 'ruscii', 'cp-gr', 'iso-8859-2', 'cp65001', 'greek', 'iso8859-10', 'iso2022-jp-1', 'iso8859-7', 'csptcp154', 'shift-jisx0213', 'iso-8859-6', 'cp1026', 'big5hkscs', 'cp875', 'utf32', 'hz-gb', 'cyrillic', '949', 'csiso58gb231280', 'cp1250', 'utf-16-be', '936', 'utf-32-le', 'shift-jis-2004', 'cp1255', 'cp437', 'cp737', 'ms949', 'ksc5601', 'unicode-1-1-utf-7', 'mac-greek', '861', 'iso8859-16', 'utf-8-sig', 'iso2022jp-2', 'ebcdic-cp-be', 'l7', 'cp865', 'iso8859-6', 'cp1006', 'latin10', 'cp869', 'windows-1255', 'eucjis2004', 'macintosh', 'ebcdic-cp-ch', 'iso-2022-jp-2004', 'johab', 'cp874', 'ibm869', 'uhc', '850', 'ebcdic-cp-he', 'iso-2022-jp-1', '1125', 'cp866u', 'utf-16', 'ks-c-5601'}

Allowed character encodings. This list is based on the list of standard encodings here:

Python specific encodings and binary transformations (eg zlib decompression) are excluded to avoid zip bomb style attacks.

fresco.util.http.CHUNK_SIZE = 8192

Data chunk size to read from the input stream (wsgi.input)

class fresco.util.http.FileUpload(filename, headers, fileob)

Represent a file uploaded in an HTTP form submission

save(fileob)

Save the upload to the file object or path fileob

Parameters:fileob – a file-like object open for writing, or the path to the file to be written
class fresco.util.http.HTTPMessage(policy=Compat32())

Represent HTTP request message headers

exception fresco.util.http.MissingContentLength(*args, **kwargs)

No Content-Length header given

class fresco.util.http.ParsedContentType(content_type, encoding, params)
content_type

Alias for field number 0

encoding

Alias for field number 1

params

Alias for field number 2

exception fresco.util.http.TooBig(*args, **kwargs)

Request body is too big

fresco.util.http.encode_multipart(data=None, files=None, charset='UTF-8', **kwargs)

Encode data using multipart/form-data encoding, returning a tuple of (<encoded data>, <environ items>).

Parameters:
  • data – POST data to be encoded, either a dict or list of (name, value) tuples.
  • charset – Encoding used for any unicode values encountered in data
  • files – list of (name, filename, content_type, data) tuples. data may be either a byte string, iterator or file-like object.
  • kwargs – other data items as keyword arguments
Returns:

a tuple of (<encoded_data>, <environ_items>), where encoded_data is a BytesIO object and environ is a dict containing the Content-Type and Content-Length headers encoded for inclusion in a WSGI environ dict.

fresco.util.http.get_body_bytes(environ, max_size=16384)

Read a single message body from environ[‘wsgi.input’], returning a bytes object.

fresco.util.http.get_content_type_info(environ, default_type='application/octet-stream', default_encoding='iso-8859-1')

Read and parse the Content-Type header and return a ParsedContentType object.

fresco.util.http.parse_body(environ, fp, default_charset=None, max_size=16384)
Parse the message
(payload as a byte string, content-type, encoding)
fresco.util.http.parse_header(header, ie_workaround=False, _broken_encoding_sniffer=<built-in method search of _sre.SRE_Pattern object>)

Given a header, return a tuple of (value, [(parameter_name, parameter_value)]).

Example usage:

>>> parse_header("text/html; charset=UTF-8")
('text/html', {'charset': 'UTF-8'})
>>> parse_header("multipart/form-data; boundary=-------7d91772e200be")
('multipart/form-data', {'boundary': '-------7d91772e200be'})
fresco.util.http.parse_multipart(fp, boundary, default_charset, max_size, ie_workaround=True)

Parse data encoded as multipart/form-data. Generate tuples of:

(<field-name>, <data>)

Where data will be a string in the case of a regular input field, or a FileUpload instance if a file was uploaded.

Parameters:
  • fp – input stream from which to read data
  • boundary – multipart boundary string, as specified by the Content-Disposition header
  • default_charset – character set to use for encoding, if not specified by a content-type header. In practice web browsers don’t supply a content-type header so this needs to contain a sensible value.
  • max_size – Maximum size in bytes for any non file upload part
  • ie_workaround – If True (the default), enable a work around for IE’s broken content-disposition header encoding.
fresco.util.http.parse_parameters(s, preserve_backslashes=False)

Return s parsed as a sequence of semi-colon delimited name=value pairs.

Example usage:

>>> from fresco.util.http import parse_parameters
>>> parse_parameters('foo=bar')
{'foo': 'bar'}
>>> parse_parameters('foo="bar\""')
{'foo': 'bar"'}

The preserve_backslashes flag is used to preserve IE compatibility for file upload paths, which it incorrectly encodes without escaping backslashes, eg:

Content-Disposition: form-data; name="file"; filename="C:       mp\Ext.js"

(To be RFC compliant, the backslashes should be doubled up).

fresco.util.http.parse_post(environ, fp, default_charset=None, max_size=16384, max_multipart_size=2097152, ie_workaround=True)

Parse the contents of an HTTP POST request, which may be either application/x-www-form-urlencoded or multipart/form-data encoded.

Returned items are either tuples of (name, value) for simple string values or (name, FileUpload) for uploaded files.

Parameters:
  • max_multipart_size – Maximum size of total data for a multipart form submission
  • max_size – The maximum size of data allowed to be read into memory. For a application/x-www-form-urlencoded submission, this is the maximum size of the entire data. For a multipart/form-data submission, this is the maximum size of any individual field (except file uploads).
fresco.util.http.parse_querystring(data, charset=None, strict=False, keep_blank_values=True, pairsplitter=<built-in method split of _sre.SRE_Pattern object>)

Return (key, value) pairs from the given querystring:

>>> list(parse_querystring('green%20eggs=ham;me=sam+i+am'))
[(u'green eggs', u'ham'), (u'me', u'sam i am')]
Parameters:
  • data – The query string to parse.
  • charset – Character encoding used to decode values. If not specified, fresco.DEFAULT_CHARSET will be used.
  • keep_blank_values – if True, keys without associated values will be returned as empty strings. if False, no key, value pair will be returned.
  • strict – if True, a ValueError will be raised on parsing errors.
fresco.util.http.parse_value(s)

Return s parsed as a RFC2045 value, ie:

value := token / quoted-string

Example usage:

>>> from fresco.util.http import parse_value
>>> parse_value('foo')
'foo'
>>> dequote('"foo"')
'foo'

fresco.util.io

Utilities for wrapping IO streams. These are used internally by fresco when parsing wsgi request input streams.

class fresco.util.io.DelimitedInput(raw, delimiter, consume_delimiter=True)

Wrap a PutbackInput to read as far as a delimiter (after which subsequent reads will return empty strings, as if EOF was reached)

Examples:

>>> from io import BytesIO
>>> s = BytesIO(b'one--two--three')
>>> p = PutbackInput(s)
>>> DelimitedInput(p, b'--').read()
'one'
>>> DelimitedInput(p, b'--').read()
'two'
>>> DelimitedInput(p, b'--').read()
'three'
>>> DelimitedInput(p, b'--').read()
''
close()

Don’t close the underlying stream unless it has been fully consumed

read(size=-1)

Return up to size bytes of data from the stream until EOF or delimiter is reached.

readline(size=-1)

Read a single line of up to size bytes from the input stream, or up to delimiter if this is encountered before a complete line is read.

class fresco.util.io.ExpandableOutput(bufsize=16384)

Write-only output object.

Will store data in a BytesIO, until more than bufsize bytes are written, at which point it will switch to storing data in a real file object.

getstorage()

Return the underlying stream (either a BytesIO or file object)

switch_to_file_storage()

Switch the storage backend to an instance of TemporaryFile.

write_file(data)

write, optimized for the TemporaryFile backend

write_stringio(data)

write, optimized for the BytesIO backend.

class fresco.util.io.PutbackInput(raw)

Wrap a file-like object to allow data read to be returned to the buffer.

Only supports serial read-access, ie no seek or write methods.

Example:

>>> from io import BytesIO
>>> s = BytesIO(b"the rain in spain\nfalls mainly\non the plain\n")
>>> p = PutbackInput(s)
>>> line = p.readline()
>>> line
'the rain in spain\n'
>>> p.putback(line)
>>> p.readline()
'the rain in spain\n'
peek(size)

Peek ahead size bytes from the stream without consuming any data

putback(data)

Put data back into the stream

read(size=-1)

Read and return up to size bytes.

readline(size=-1)

Read a single line of up to size bytes.

class fresco.util.io.ReadlinesMixin

Mixin that defines readlines and the iterator interface in terms of underlying readline method.

class fresco.util.io.SizeLimitedInput(raw, length)

Wrap an IO object to prevent reading beyond length bytes.

Example:

>>> from io import BytesIO
>>> s = BytesIO(b"the rain in spain\nfalls mainly\non the plain\n")
>>> s = SizeLimitedInput(s, 24)
>>> len(s.read())
24
>>> s.seek(0)
0L
>>> s.read()
'the rain in spain\nfalls '
>>> s.seek(0)
0L
>>> s.readline()
'the rain in spain\n'
>>> s.readline()
'falls '
check_available(requested)

Return the minimum of requested and the number of bytes available in the stream. If requested is negative, return the number of bytes available in the stream.

read(size=-1)

Return up to size bytes from the input stream.

readline(size=-1)

Read a single line of up to size bytes from the input stream.

seek(pos, whence=0)

Seek to position pos. This is a wrapper for the underlying IO’s seek method.

tell()

Return the position of the file pointer in the stream.

fresco.util.wsgi

Utilities for interfacing with WSGI

fresco.util.wsgi.environ_to_unicode(s, enc='UTF-8')

Decode a WSGI environ value to a unicode string

fresco.util.wsgi.unicode_to_environ(s, enc='UTF-8')

Return a unicode string encoded for a WSGI environ value

In python 2 this function returns s encoded using the specified encoding

In python 3 this returns a ‘bytes-as-unicode’ string:

  • encode s using the specified encoding (eg utf8)
  • decode the resulting byte string as latin-1
fresco.util.wsgi.environ_to_bytes(s)

Decode a WSGI environ value to a bytes object

fresco.util.wsgi.bytes_to_environ(s)

Encode a byte string to a WSGI environ value

For Python 2, this simply returns s. For Python 3 this returns a ‘bytes-as-unicode’ string.

class fresco.util.wsgi.StartResponseWrapper(start_response)

Wrapper class for the start_response callable, allowing middleware applications to intercept and interrogate the proxied start_response arguments.

Synopsis:

>>> def my_wsgi_app(environ, start_response):
...     start_response('200 OK', [('Content-Type', 'text/plain')])
...     return ['Whoa nelly!']
...
>>> def my_other_wsgi_app(environ, start_response):
...     responder = StartResponseWrapper(start_response)
...     result = my_wsgi_app(environ, responder)
...     print "Got status", responder.status
...     print "Got headers", responder.headers
...     responder.call_start_response()
...     return result
...
>>> from flea import Agent
>>> result = Agent(my_other_wsgi_app).get('/')
Got status 200 OK
Got headers [('Content-Type', 'text/plain')]

See also Response.from_wsgi, which takes a wsgi callable, environ and start_response, and returns a Response object, allowing the client to further interrogate and customize the WSGI response.

Note that it is usually not advised to use this directly in middleware as start_response may not be called directly from the WSGI application, but rather from the iterator it returns. In this case the middleware may need logic to accommodate this. It is usually safer to use Response.from_wsgi, which takes this into account.

call_start_response()

Invoke the wrapped WSGI start_response function.

class fresco.util.wsgi.ClosingIterator(iterable, *close_funcs)

Wrap a WSGI iterator to allow additional close functions to be called on application exit.

Synopsis:

>>> class filelikeobject(object):
...
...     def read(self):
...         print "file read!"
...         return ''
...
...     def close(self):
...         print "file closed!"
...
>>> def app(environ, start_response):
...     f = filelikeobject()
...     start_response('200 OK', [('Content-Type', 'text/plain')])
...     return ClosingIterator(iter(f.read, ''), f.close)
...
>>> from flea import Agent
>>> m = Agent(app).get('/')
file read!
file closed!
close()

Call all close functions listed in *close_funcs.

next()

Return the next item from iterator

fresco.util.wsgi.make_environ(url='/', environ=None, wsgi_input=b'', middleware=True, **kwargs)

Return a WSGI environ dict populated with values modelling the specified request url and data.

Parameters:
  • url – The URL for the request, eg /index.html or /search?q=foo. Note that url must be properly URL encoded.
  • environ – values to pass into the WSGI environ dict
  • wsgi_input – The input stream to use in the wsgi.input key of the environ dict
  • middleware – If False the middleware stack will not be invoked. Disabling the middleware can speed up the execution considerably, but it will no longer give an accurate simulation of a real HTTP request.
  • kwargs – additional keyword arguments will be passed into the WSGI request environment