| Age | Commit message (Collapse) | Author |
|---|
|
utilities, as well as the tests themselves. File became too large
|
|
lock is not actually released or they are not actually notifyied, staying in a beautysleep. This glitch is probably caused by some detail not treated correctly in the thread python module, which is something we cannot fix. It works most of the time as expected though - maybe some cleanup is not done correctly which causes this
|
|
well as concurrency issues. Now it works okay, but the thread-shutdown is still an issue, as it causes incorrect behaviour making the tests fail. Its good, as it hints at additional issues that need to be solved. There is just a little more left on the feature side, but its nearly there
|
|
closed only when there is no one else writing to it. This assures that all tasks can continue working, and put their results accordingly. Shutdown is still not working correctly, but that should be solvable as well. Its still not perfect though ...
|
|
readers and writers, a ready is not connected to its writer anymore. This changes the refcounting of course, which is why the auto-cleanup for the pool is currently broken.
The benefit of this are faster writes to the channel, reading didn't improve, refcounts should be clearer now
|
|
fully deterministic as tasks still run into the problem that they try to write into a closed channel, it was closed by one of their task-mates who didn't know someone else was still computing
|
|
the same and different pools
|
|
allows the pool to work as expected. Many more tests need to be added, and there still is a problem with shutdown as sometimes it won't kill all threads, mainly because the process came up with worker threads started, which cannot be
|
|
reference counting mechanism, causing references to the pool to be kepts via cycles
|
|
channel implementation, one of which is used as base by the Pool Read channel, releasing it of the duty to call these itself. The write channel with callback subclass allows the transformation of the item to be written
|
|
makes it easier to constomize
pool: in serial mode, created channels will be serial-only, which brings 15% of performance
|
|
and it runs faster as well, about 2/3 of the performance we have when being in serial mode
|
|
Condition implementation, related to the notify method not being treadsafe. Although I was aware of it, I missed the first check which tests for the size - the result could be incorrect if the whole method wasn't locked.
Testing runs stable now, allowing to move on \!
|
|
one more level of indirection. Clearly this not good from a design standpoint, as a Condition is no Deque, but it helps speeding things up which is what this is about. Could make it a hidden class to indicate how 'special' it is
|
|
queue: Queue now derives from deque directly, which safes one dict lookup as the queue does not need to be accessed through self anymore
pool test improved to better verify threads are started correctly
|
|
thread-safe, causing locks to be released multiple times. Now it runs very fast, and very stable apparently.
Now its about putting previous features back in, and studying their results, before more complex task graphs can be examined
|
|
events only with its queue, with boosts performance into brigt green levels
|
|
the time instead of reusing its own one, it was somewhat hard to manage its state over time and could cause bugs. It works okay, but it occasionally hangs, it appears to be an empty queue, have to gradually put certain things back in, although in the current mode of operation, it should never have empty queues from the pool to the user
|
|
thing as the task deletes itself too late - its time for a paradigm change, the task should be deleted with its RPoolChannel or explicitly by the user. The test needs to adapt, and shouldn't assume anything unless the RPoolChannel is gone
|
|
task class
|
|
the wrong spot. The channel is nothing more than an adapter allowing to read multiple items from a thread-safe queue, the queue itself though must be 'closable' for writing, or needs something like a writable flag.
|
|
at least with my totally overwritten version of the condition - the previous one was somewhat more stable it seems. Nonetheless, this is the fastest version so far
|
|
didn't seem necessary - its a failure, something is wrong - performance not much better than the original one, its depending on the condition performance actually, which I don't get faster
|
|
unnecessary tasks to be scheduled as we keep track of how many items will be produced for the task at hand. This introduces additional locking, but performns well in multithreaded mode. Performance of the master queue is still a huge issue, its currently the limiting factor, as bypassing the master queue in serial moode gives 15x performance, wich is what I would need
|
|
still inconsistencies that need to be fixed, but it already improved, especially the 4-thread performance which now is as fast as the dual-threaded performance
|
|
for up to 2 threads, but 4 are killing the queue
|
|
least in tests, and with multiple threads. There is still an sync bug in regard to closed channels to be fixed, as the Task.set_done handling is incorrecft
|
|
changing tasks
Now processing more items to test performance, in dual-threaded mode as well, and its rather bad, have to figure out the reason for this, probably gil, but queues could help
|
|
multiple connected pools
Reduced waiting time in tests to make them complete faster
|
|
single task for now, but next up are dependent tasks
|
|
related to our channel closed flag, which is the only way not to block forever on read(0) channels which were closed by a thread 'in the meanwhile'
|
|
chunking test. Next up, actual async processing
|
|
including own tests, their design improved to prepare them for some specifics that would be needed for multiprocessing support
|
|
is handled by the task system
graph: implemented it including test according to the pools requirements
pool: implemented set_pool_size
|
|
while going. Tests will be written soon for verification, its still quite theoretical
|
|
going on. The default implementation uses threads, which ends up being nothing more than async, as they are all locked down by internal and the global interpreter lock
|
|
to do with the object db. If that really works the way I want, it will become an own project, called async
|
|
inter-dependent tasks
|
|
Git-Python. I have the feeling it can do much good here :)
|
|
restructured odb tests, they are now in an own module to keep the modules small
|
|
will always be compressed if generated by the system ( even future memory db's will compress it )
loose db: implemented direct stream copy, indicated by a sha set in the IStream, including test. This will be the case once Packs are exploded for instance
|
|
for streams starts to show up, but its not yet there
|
|
but next there will have to be more through testing
|
|
multi-threading implementation of all odb functions
|
|
|
|
parsing which truncated newlines although it was ilegitimate. Its up to the reader to truncate therse, nowhere in the git code I could find anyone adding newlines to commits where it is written
Added performance tests for serialization, it does about 5k commits per second if writing to tmpfs
|
|
|
|
missing ) and added performance tests which are extremely promising
|
|
efficiently considering that it copies string buffers all the time
|
|
|
