New Contributors to HugeCollections

By

What is the HugeCollection library

The objectives of HugeCollections for the first release are fairly ambitious.
  • Scale to massive collections i.e. sizes much larger than 2 billion without significant heap foot print or GC impact. i.e. using heap-less memory
  • Be faster and more efficient that using plain JavaBeans with an ArrayList or Map and a vector or unordered_map in C++.
  • Support durability (transparently saved and loaded from disk)
  • Support thread safety (with no overhead if not required) and using multiple threads implicitly. i.e. large operations are automatically distributed.
  • Be faster and more efficient than using a database. Transactions will NOT be in this release. 
  • Support in one application what might have to be distributed otherwise.
  • Dynamic code generation as required (no need to pre-generate code in the build)
A prototype has been built which shows these objectives are possible, however to turn this library in to a usable release, will take some help.

Note: I am open to people working on the things they find most interesting, even if this means some areas are more developed than others.

To have a look at the code start with http://vanilla-java.googlecode.com/svn/sandbox/ which you can also checkout.

Myself

Peter Lawrey - I have been working with Java for 12 years and on high performance systems for 15 years. I have worked at Investment Banks, a prop trading firm and Sun Microsystems.

Two new contributors

Rob Austin currently contracting for a large investment bank, working on a low latency pricing and trading platform. Over 10 years of Java experience.

Costantino Cerbo is a certified Java developer (SCJP and SCBCD) and software consultant with more than 6 years experience. He has worked for one of the Italy's largest banks. He is an Italian native speaker, fluent in German and strong in English (TOEFL: 607 points).

More contributors welcome

I am looking for additional contributors to....
- document the high level approach used and its advantages.
- proper documentation and pier review of the design.
- code review of the hand coded collections for list and hash map (These are templates for the auto-generated classes)- tool to assist the conversion of the template into the code generation.
- Test cases for correct functionality.
- Performance and scalability tests.
- Comparison tests for performance and scalability.
- a comparison with the features of similar products like ehCache BigMemory, javolution, trove4j (also find other products worth comparing) and C++.

- Documentation and blog of the comparison.
- Examine thread safety support and tests.
- Auto multi-threading for filter() and visit() methods.
- Examine integration and examples of use in JVM languages like Scala, Groovy, Jython, JRuby and see what support can be given. i.e. are there simple things which can be done to make it simpler/more natural to use.
- Produce a JTable GUI demo with one billion rows.

Later
- Add queue/dispatcher support.
- Add sorted index (a la TreeMap or map in C++) support.
- Add non-unique indexes.
- Simple remote support. RMI/RPC
- Distributed copies of data and partitioning.

Any suggestions welcome.

My email

You can contact me as peter.lawrey on gmail.

Comments

  1. Thomas Mueller20 September 2011 at 08:49

    Maybe I could help support CSV import/export, a SQL query engine, JDBC support, and a simple web frontent. I would write a custom table implementation in my H2 database to allow to access such collections. This wouldn't necessarily be fast, but convenient (running ad-hoc queries, import and export).

    ReplyDelete
  2. Peter Lawrey20 September 2011 at 09:28

    @Thomas Mueller, That sounds like a good suggestion. CSV import and export would be particularly useful. How easy would implementing the other features be?

    I will add my email address

    ReplyDelete
  3. Aleksandr26 September 2011 at 10:19

    How about list comprehensions?

    ReplyDelete
  4. Peter Lawrey26 September 2011 at 11:59

    @Aleksandr Panzin, That does sound useful. esp to support partitioned data.

    ReplyDelete
  5. Peter Lawrey26 September 2011 at 12:02

    @Aleksandr Panzin, I have added this as an issue. To support lists and maps.

    ReplyDelete

Post a Comment

Popular posts from this blog

Java is Very Fast, If You Don’t Create Many Objects

By
Image
  You still have to watch how many objects you create. This article looks at a benchmark passing events over TCP/IP at 4 billion events per minute using the net.openhft.chronicle.wire.channel package in Chronicle Wire and why we still avoid object allocations..  One of the key optimisations is creating almost no garbage. Allocation is a very cheap operation and collection of very short-lived objects is also very cheap. Does this really make a difference? What difference does one small object per event (44 bytes) make to the performance in a throughput test where GC pauses are amortised? While allocation is as efficient as possible, it doesn’t avoid the memory pressure on the L1/L2 caches of your CPUs and when many cores are busy, they are contending for memory in the shared L3 cache.  Results Benchmark on a Ryzen 5950X with Ubuntu 22.10. JVM Vendor, Version No objects Throughput, Average Latency* One object per event Throughput, Average Latency* Azul Zulu 1.8.0_322 60.6 M event/s, 528
Read more

System wide unique nanosecond timestamps

By
A Unique Identifier can be very useful for tracing. Those ids are even more useful when they contain a high-resolution timestamp.  Not only do they record the time of an event, but if unique can help trace events as they pass through the system. Such unique timestamps however can be expensive depending on how they are implemented.   This post explores a lightweight means of producing a unique, monotonically increasing system-wide nano-second resolution timestamp available in our open-source library. Uses for Unique Identifiers Unique identifiers can be useful to associate with a piece of information so that information can be referred to later unambiguously. This could be an event, a request, an order id, or a customer id. They can naturally be used as a primary key in a database or key/value store to retrieve that information later. One of the challenges of generating these identifiers is avoiding creating duplicates while not having an increasing cost.  You could record every identi
Read more

Comparing Approaches to Durability in Low Latency Messaging Queues

By
Image
  A significant feature of Chronicle Queue Enterprise is support for TCP replication across multiple servers to ensure the high availability of application infrastructure. I generally believe that replicating data to a secondary system is faster than syncing to disk, assuming the round trip network delay wasn’t high due to quality networks and co-located redundant servers. This is the first time I have benchmarked it with a realistic example. Little’s Law and Why Latency Matters In many cases, the assumption is that the latency won't be a problem as long as throughput is high enough. However, latency is often a key factor in why the throughput isn’t high enough. Little’s law states, “ the long-term average number  L  of customers in a  stationary  system is equal to the long-term average effective arrival rate  λ  multiplied by the average time  W  that a customer spends in the system”. In computer terminology, the level of concurrency or parallelism a system has to support must be
Read more