Adds a nice TBinarySerializer class to make it simple to convert between PHP objects and serialized strings using the Binary protocol.
Patch: Radu Marin
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1021521 13f79535-47bb-0310-9956-ffa450edef68
Move processing code out of main try block and created individual
try/catch's for closing down the client and it's transports (to avoid
issues with throwing exceptions from a catch block).
Also converted to using GlobalOutput instead of cerr.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005171 13f79535-47bb-0310-9956-ffa450edef68
The T_DEBUG* and T_ERROR* macros used preprocessor stringification to
stringify the format string argument. This was weird and unintuitive.
With the old behavior:
- Quotes surrounding the format string were included in the message:
T_DEBUG("this is a test") --> expanded to "\"this is a test\""
- Backslashes in the string are escaped so they print literally:
T_DEBUG("foo\nbar") --> expanded to "\"foo\\nbar\""
- Standard fixed-width integer format macros don't work:
T_DEBUG("x: %" PRIi64, x) --> expanded to "\"x: %\" PRIi64"
The last item is particularly problematic, since it prevents 64-bit
values from being logged portably.
With the new code, the following will no longer compile:
T_DEBUG(this is my log message: %d, 5)
I don't think that is a bad thing, though.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005170 13f79535-47bb-0310-9956-ffa450edef68
Add two improvements to memory management in TNonblocking server:
- Separate the receive code into two distinct states: one for receiving
the frame header and one for the frame content. This allows us to
size the initial read buffer based on the initial frame size, rather
than allocating an arbitrary amount of memory before reading the
header.
- Allow setting the initial write buffer size based on the application's
expected response size.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005169 13f79535-47bb-0310-9956-ffa450edef68
Although there is a constructor that allows a TMemoryBuffer to be
created with an internally-allocated buffer of a desired size, there was
no way for resetBuffer to obtain analogous behavior. This change causes
the appropriate constructor to be used.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005168 13f79535-47bb-0310-9956-ffa450edef68
When malloc/realloc fail, we've typically just thrown a TException. This
allows a server that should simply crash when out of memory to survive
in a strage state, with various bad consequences. Instead, we should
throw bad_alloc and just not catch it (or if we decide to, be very
careful to respond properly).
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005167 13f79535-47bb-0310-9956-ffa450edef68
Previously, if we had a new[] failure when growing a TFramedTransport
write buffer, we would leave the buffer in an invalid state (wBufSize_
would reflect the desired size, rather than the actual size). Now, we
make no change to any member variables if new[] fails.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005166 13f79535-47bb-0310-9956-ffa450edef68
Previously, if we had a realloc failure when growing a TMemoryBuffer, we
would leave the buffer in an invalid state (bufferSize_ would reflect
the desired size, rather than the actual size). Now, we make no change
to any member variables if realloc fails.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005165 13f79535-47bb-0310-9956-ffa450edef68
Add methods to TNonblockingServer to set the maximum size of idle read
and write buffers and the check interval (in calls). When checked, if
the buffers are larger than the configured maximum, they will be resized
down the to maximum size.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005164 13f79535-47bb-0310-9956-ffa450edef68
r750153 caused TNonblockingServer to reset its buffers every 512 calls.
A more configurable version was developed internally, so I'm reverting
this rev first to avoid conflicts.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005163 13f79535-47bb-0310-9956-ffa450edef68
The boost test framework has changed significantly from boost 1.34 to
1.37. Quite a few new features have been added, and some annoying bugs
have been fixed.
This change now builds the thrift tests against boost 1.37, and updates
them to use some of the newer features.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005162 13f79535-47bb-0310-9956-ffa450edef68
- TBufferedTransport::borrow() could block if not enough data was
available. Now it returns NULL immediately in this case, like all
other transports.
- TBufferedTransport::read() could block some data was available in the
readahead buffer, but not enough to satisfy the request. It would
attempt to call read() on the underlying transport, but this might
block. Now it just returns the remaining data in the readahead
buffer. The caller is responsible for calling read() again to get the
rest of the data they want.
- TFrameTransport::read() threw an exception if read() on the underlying
transport returned 0 when looking for a frame header. Now
TFrameTransport::read() returns 0, too. (It still throws an exception
if the underlying transport returns 0 after a partial frame or frame
header has been read.)
- TFDTransport::read() threw an exception on EINTR. Now it retries up
to 5 times, similarly to the way TSocket::read() behaves.
- TZlibTransport::read() could block when less data than was requested
is available. Now it only calls read() on the underlying transport
when it would otherwise have nothing to return.
This does mean that TZlibTransport::read() now often returns less data
than is actually available at the time. This required updating
several of the ZlibTest tests to use readAll() instead of read(),
since they previously assumed read() would return all available data.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005161 13f79535-47bb-0310-9956-ffa450edef68
Add tests that check to see whether or not read() and borrow() block
when called with a length larger than the amount of data currently
available.
At the moment, not all of the transports behave the same way. I believe
the desired behavior is:
When M bytes are available, and 0 < M < N:
- read(N): return M bytes immediately
- borrow(N): return NULL immediately
When 0 bytes are available:
- read(N): In this case, it is acceptable either to immediately return
0, or to block until some data is available. If the transport
blocks, it returns immediately when some date becomes available,
even if less than N bytes are available.
- borrow(N): return NULL immediately
- The borrow() tests fail when using TBufferedTransport.
TBufferedTransport incorrectly blocks until the amount of data
requested is available.
- test_read_none_available() fails when using TFramedTransport.
Calling read() on a TFramedTransport when no data is available throws
an exception instead of returning 0.
- test_read_none_available() fails when using TFDTransport. This is
partly just an artifact of the fact that I use SIGALRM as part of this
test. Unlike TSocket, TFDTransport doesn't retry after EINTR.
- test_read_part_available() fails when using TZlibTransport around a
transport that has blocking read() behavior. TZlibTransport::read()
loops calling read() on the underlying transport. It should probably
break out of the loop and return to the caller as soon as it has
uncompressed any data, even if it is less than requested and more
might be available. Once some data has been uncompressed,
TZlibTransport cannot risk calling read() again since it might block.
Will commit fixes for these separately.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005160 13f79535-47bb-0310-9956-ffa450edef68
UpdateS TransportTest so that the wrapper transports
(TBufferedTransport, TFramedTransport, TZlibTransport) are tested with a
wider variety of inner transports. Previously they were only tested
using TMemoryBuffer. Now all other transports are also tested wrapped
inside each of these transports.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005159 13f79535-47bb-0310-9956-ffa450edef68
Reduce the default test buffer sizes by about 30x, reducing the time it
takes to run TransportTest from about 1 minute to about 2 seconds. I
added a --size-multiplier argument that can be used to adjust the sizes
of all test buffers, so developers can still run with large buffer sizes
when desired.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005157 13f79535-47bb-0310-9956-ffa450edef68
Previously flush() had race conditions that could cause it to return
before all data had actually been flushed to disk. Now the writer
makes sure both buffer queues have been flushed when forceFlush_ is set.
Also, flush() did not wake up the writer thread, so it normally had to
wait for the writer thread to wake up on its own time. (By default,
this could take up to 3 seconds.)
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005156 13f79535-47bb-0310-9956-ffa450edef68
Several of the TFileTransportTest tests check wall clock time to make
sure the writer thread processes operations quickly enough, and isn't
hanging. However, this can easily result in false failures if we don't
get enough processor time. This commit makes a few changes to reduce
the number of these failures.
- No longer fail if a single destructor call takes more than 500us.
We only require 90% of the calls to complete in 500us. No call may
take more than 100ms, though. With this change, the test passes most
of the time now, even while an "fbmake opt" task is running in
parallel.
- In the flush_max_us tests, make sure the writer thread is started
before we start recording timing. Otherwise, creating the thread
could take long enough to throw off the numbers for the first fsync()
call.
Also tested with the pthread_cond_signal() in the TFileTransport
destructor commented out, to make sure the test still fails properly
when the destructor takes too long.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005155 13f79535-47bb-0310-9956-ffa450edef68
Set closing_ to true before we wake up the writer thread in the
destructor. This way the writer thread flushes the data and exits when
it is woken up.
Previously the writer thread could end up going through 2 full timeout
cycles before exiting. The writer thread could wake up, notice it has
nothing to do since closing_ is not set, and immediately go back to
sleep. The destructor would then proceed to call flush(), which would
wait for the writer thread to wake up (1 full time out) and clear the
forceFlush_ flag. After flush returns, the destructor would set
closing_. It could take the writer thread another full timeout to wake
up again and process this flag.
There were also some points where the worker threads would detect the
closing_ state in their loops and automatically close the file, but did
not zero out the file descriptor, then the destructer attempted to close
the same file. Fix by simply zeroing out the fd_ at these points.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005154 13f79535-47bb-0310-9956-ffa450edef68
Previously, the TFileTransport writer thread behaved as follows:
while true:
wait for main thread to notify new data in enqueueBuffer_
swap(enqueueBuffer_, dequeueBuffer_)
write out everything in dequeueBuffer_
Now the behavior is:
while true:
if enqueueBuffer_ is empty
wait for main thread to notify new data in enqueueBuffer_
swap(enqueueBuffer_, dequeueBuffer_)
write out everything in dequeueBuffer_
The old behavior had a couple problems:
- Writes that arrived while the writer thread was writing
dequeueBuffer_ wouldn't get processed immediately. The writer thread
would always wait until another write occurred after it started its
condition variable wait, or until it timed out (3 seconds by default).
- If the main thread was writing fast enough to fill up enqueueBuffer_
while the writer thread was still writing out dequeueBuffer_, it would
block the next write call until the writer thread swapped the buffers.
Unfortunately, the writer thread waits to do this until it the main
thread notifies it of another write. This deadlock is only broken by
the 3 second timeout. Performance then tanks, since the writer thread
now always sleeps 3 seconds each time around the loop.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005153 13f79535-47bb-0310-9956-ffa450edef68
Now that TZlibTransport::flush() behaves the same way as other
transports, there is no need to distinguish between RPC and standalone
behavior for TZlibTransport.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005152 13f79535-47bb-0310-9956-ffa450edef68
Previously, TZlibTransport::flush() finished the zlib stream, so calling
write() after flush() would result in an error. Now it just flushes the
data, without finishing the stream. A new TZlibTransport::finish()
function has been added to finish the stream.
This breaks compatibility. I'm aware of anyone using this code outside
of Facebook, though.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005151 13f79535-47bb-0310-9956-ffa450edef68
This way we no longer have to have a huge hard-coded list of numbers in
the source code. The distribution is randomly generated for each run.
(Although the --seed argument can be used for repeatablity.)
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005150 13f79535-47bb-0310-9956-ffa450edef68
Previously, ZlibTest read a file from disk to get data to test with.
It would fail unless gen-cpp/DebugProtoTest_types.tcc was present in the
current directory and was at least 32kB long.
Now ZlibTest simply generates 3 separate buffers to test with. The
first buffer is just all "a"s, the second is some random sequential
runs, and the third is completely random. They usually seem to have
compression ratios of around 315:1, 4:1, and 1:1, respectively.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005148 13f79535-47bb-0310-9956-ffa450edef68
Tests various transport types by writing data, and verifying it can be
read back successfully. Tests both virtual calls (accessed as
TTransport*) and non-virtual calls (accessed as the appropriate pointer
type, or as TBufferBase* when appropriate).
This is similar to some of the tests already performed in
TMemoryBufferTest and ZlibTest.cpp. However, this tests a few more
transport types, and it interleaves read and write calls more heavily.
(It currently exposes some bugs in flush() in a couple transports.) It
also exercises both the virtual and non-virtual code paths, now that
read() and write() only result in virtual calls when invoked on a
TTransport*.
TFileTransport currently has several race condition bugs, so most of the
TFileTransport tests ususally fail. It also has some performance bugs,
causing the TFileTransport tests to take a long time. Will fix those
issues separately.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005146 13f79535-47bb-0310-9956-ffa450edef68
Modify TNonblockingServer to use TSocket for I/O and support server
event handlers; this enables TClientInfo to function with a minor change
to the processing loop.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005145 13f79535-47bb-0310-9956-ffa450edef68
Add some profiling code to track when potentially unnecessary virtual
calls are made in the thrift C++ serialization and deserialization code.
This can be used to help service implementors determine which places in
their code should be updated to use an appropriate thrift template
class.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005140 13f79535-47bb-0310-9956-ffa450edef68
Add the ability for Thrift servers to monitor client connections. It is
activated by #including server/TClientInfo.h and creating 1) a
TClientInfoCallHandler passed to the processor with setEventHandler()
and 2) a TClientInforServerHandler passed to the server with
setServerEventHandler().
The result vector, showing active connections, provides client address
and the thrift call it is executing (or last executed), the time
connected, and the number of calls made since connection.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005139 13f79535-47bb-0310-9956-ffa450edef68
When the "templates" option is passed to the C++ generator, it now emits
templatized versions of the client and processor. Generated types emit
templatized read() and write() functions.
This allows the generated code to invoke the correct non-virtual
TTransport and TProtocol implementations, resulting in faster
serialization and deserialization.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005138 13f79535-47bb-0310-9956-ffa450edef68
The STL specializes vector<bool> to store the values as individual bits, rather
than bools. Therefore, when using a Thrift list<bool>, readBool() gets invoked
not with a bool&, but with a std::vector<bool>::reference.
TProtocol does provide a readBool(std::vector<bool>::reference) implementation.
However, almost all TProtocol subclasses defined only readBool(bool&), which
hides the other overloaded versions of readBool(). As a result, the code
worked only when accessing TProtocol objects via a "TProtocol*", and not
directly via the subclass type. When using C++ templates, protocol objects do
get invoked via pointers to the subclass type, causing compile failures when
std::vector<bool> is used.
This change updates the various TProtocol implementations to also provide
readBool(std::vector<bool>::reference).
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005137 13f79535-47bb-0310-9956-ffa450edef68
Convert TBinaryProtocol and TCompactProtocol to template classes, taking
the transport class as a template parameter. This allows them to make
non-virtual calls when using the template, improving serialization
performance.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005136 13f79535-47bb-0310-9956-ffa450edef68
Updated the thrift protocol classes to use non-virtual calls for most
functions. The correct implementation is determined at compile time via
templates now. Only the base TProtocol class falls back to using
virtual function calls.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005135 13f79535-47bb-0310-9956-ffa450edef68
Update the thrift transport classes to use non-virtual calls for most
functions. The correct implementation is determined at compile time via
templates now. Only the base TTransport class falls back to using
virtual function calls.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005134 13f79535-47bb-0310-9956-ffa450edef68
Just perform a memcpy() if all of the requested data is available in the
buffer. This improves performance a little in the common case. It has
a bigger impact with the upcoming template changes.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005133 13f79535-47bb-0310-9956-ffa450edef68
Attempt to get a pointer to the internal transport buffer before copying
onto the heap. This improves performance TFramedTransport and
TMemoryBuffer, and with TBufferedTransport if the string fits within the
buffer.
git-svn-id: https://svn.apache.org/repos/asf/incubator/thrift/trunk@1005132 13f79535-47bb-0310-9956-ffa450edef68
