Digital Bazaar Core

From DBWiki

Contents 1 Introduction 2 Documentation 3 Components 3.1 Software AutoUpdater 3.2 Cryptography Extensions 3.3 Data Format Processors 3.4 Event Delegation 3.5 Custom Java GUI components 3.5.1 Wizard Creator 3.6 Dynamic Signed Jar Class Loader 3.7 Logging Framework 3.8 Digital Bazaar Network Engine 3.8.1 Micro Web Server 3.8.2 HTTP Extensions to Micro Web Server 3.8.3 WSDL Extensions to Micro Web Server 3.8.4 SOAP Extensions to Micro Web Server 3.8.5 Live Data Stream Processing 3.9 Operation Engine 3.10 DB UPnP Client Stack 3.11 XML Processing Utilities

Introduction

The Digital Bazaar Core source code includes many core classes, tools and utilities for implementing fast, clustered, distributed processing-based web services.

Documentation

Full documentation is provided for the entire dbcore software stack. You can browse the latest documentation online.

Components

Software AutoUpdater

The Software AutoUpdater package provides a set of interfaces that can be implemented to update applications that have already been installed on user machines. These interfaces allow for an AutoUpdater engine to be dynamically loaded so that it can update itself, if necessary.

The package also contains a basic implementation of these interfaces. The basic implementation includes a script processing engine that allows files to be downloaded, verified against a checksum, and installed. An update script also has the ability to delete files and directories and run patches that have been verified against a checksum.

There is no GUI included with the basic implementation, however, event delegates are provided so that a GUI may be easily attached to monitor events as update scripts are processed as well as provide a means for user input.

Cryptography Extensions

DB core's library includes a set of tools that make using cryptography in an application, especially PKI, easier.

Because Digital Bazaar's technology relies heavily on digital signatures, several classes have been implemented to abstract away many of their details. Some of those details include the storage structure of cryptography keys.

Private and Public keys are often represented in ASN.1 (Abstract Syntax Notation #1). ASN.1 is a language that is used to describe information that is exchanged between programs -- as private, and to a much greater extent, public keys are. A common binary format used for storing ASN.1 values is called DER (Distinguished Encoding Rules). Java provides some classes for handling DER structures, however, it relies heavily on its own keystore format and implementation. This means that private and public keys cannot be stored or manipulated independently very easily.

Since Digital Bazaar requires a more "free" atmosphere for the storage of and shared use of cryptography keys between and amongst different technologies we have implemented some core classes for generating, reading, writing, and making use of such keys as stored independently in PKCS#8, X.509, and PEM format.

The KeyManager class is primarily responsible for the tasks of key generation, key reading and writing, and key format manipulation. It makes use of several helper classes: Asn1Der, Pksc8PrivateKey, and X509PublicKey to provide the handle the actual binary format details for cryptography keys.

The Cryptor class makes use of asymmetic (private and public) keys by providing a simply interface for digitally signing and verifying data and text. The Cryptor class can also encrypt and decrypt data using symmetric keys as well as produce MD5 and SHA-1 checksums.

The PrivateKeyCryptor builds upon the KeyManager and Cryptor classes by providing an interface for easily storing password-encrypted private keys both in memory and on disk.

By using these classes in combination, cryptography keys (in DSA, RSA, DES, and AES format) can be generated and put to use to sign, verify, encrypt, and decrypt data for secure communication.

Data Format Processors

Digital Bazaar's technology reads, writes, and manipulates several data formats (and is always expanding the list of supported formats). DB core's data format package contains classes for handling such formats.

DB's streaming technology has a need to recognize and populate meta data in MPEG audio streams as well as modify the actual data in those streams. Therefore classes to read, write, and manipulate ID3 tags and MPEG frames have been created. Using these classes, a stream of MPEG audio data can be parsed into frames and the data in those frames can be read and manipulated as necessary.

DB's technology also makes use of video files that are stored in AVI format. Because AVI makes use of the RIFF (Resource Interchange File Format) several classes have been implemented to handle parsing and manipulating RIFF data.

Event Delegation

After exploring several different methods of handling and raising events (and several iterations of implementations for each method), a C#-like event delegate structure was adopted by Digital Bazaar for quick creation of software events.

One of DB core's tools from its util package is the MethodInvoker class. This class functions much like C#'s Delegate class. It acts as a pointer or container for a method so that it can be invoked at a later time, for instance, as a call back after an event has occurred. The method can be invoked either in the current foreground process, or in a background process. The classes in DB core's event package make extensive use of this class.

The EventDelegate class allows a set of listeners, or classes that implement a particular call back method, to be notified of when an event occurs. A listener may be attached to the EventDelegate by specifying the method on the listener that is to be called when the event that the EventDelegate is associated with occurs. Whenever that event is raised, each listener will have its call back invoked in the same order that the listeners were attached to the EventDelegate.

This method allows for the easy creation of listener objects without the need for creating special interfaces for each kind of event or set of events that can be raised (and, at the same time, does not prevent this kind of design if it is desired). Furthermore, call back methods can be passed generic EventObjects that have a hashtable that can be populated according to whatever is necessary for the particular event.

This method of event handling is abstract enough to provide almost all of the needs of any simple event system, saving the developer some extra time and classes that would be needed otherwise.

To aid the developer in multithreaded environments, a ThreadedEventDelegate class is provided. This class uses a thread pool to send events to each listener simultaneously. This is particularly useful in GUI environments where events occur that are not meant to be processed in a special order. Often GUI components are not meant to have "dependencies," that is, they should not have to wait for other GUI components to process certain events before they are allowed to do the same. The ThreadedEventDelegate is an appropriate tool for this kind of situation.

Custom Java GUI components

DB core's gui package provides several custom Java GUI components or tools for helping in GUI development.

The most notable custom components are EditableTabPanel and the JComponentTable.

The EditableTabPanel is an extension of the TabPanel class. The TabPanel class creates a much more customizable tab-based component than is provided by Java. The TabPanel is comprised of Tab Areas and Tab Content Areas. Each Tab Area is a Swing Component that can be populated with any Component that is desired. Furthermore, the Component is truly functional, that is, if a JButton is placed in a Tab Area then it can be pressed and will fire an ActionEvent. If the Component needs to change its appearance based on a mouse over event, it will receive that mouse event and may do as it pleases. Each Tab Content Area works in the same fashion.

The EditableTabPanel's extensions include the ability to automatically include close buttons on each Tab Area so that tabs can be closed. It also includes the ability to drag Tab Areas thereby changing their order, the ability to make Tab Areas flash (i.e. like when their associated Tab Content Area changes), and some helper methods for easily setting titles and icons for Tab Areas.

The JComponentTable is an extension of Java's JTable that allows an JComponent to be placed in a JTable. This is particularly useful for putting combo boxes, radio buttons, and regular buttons into a table row.

Another custom components is a FastProgressBar that can be faster than a JProgressBar in how it redraws itself when it is being updated rapidly. It is only faster because it performs fewer and quicker redraws. It must be manually told to repaint itself by the developer, allowing the developer more control over when to repaint -- so that if several changes must be made to the progress bar, it can be repainted just once when it has finished processing every change. Furthermore, the foreground, background, and text colors for the FastProgressBar can be easily specified.

A ValidatingFileChooser is an extension of the JFileChooser that can check a FileChoiceValidator to see whether or not the choice is valid before allowing the selection. That FileChoiceValidator can popup an error message or take some other action to notify the user of why their selection was invalid.

Some classes that aid GUI development but are not full components themselves have also been implemented.

Several classes that aid in drag and drop development have been created. These classes abstract away a lot of the transfer details that must be implemented if you're using Swing's DnD API. They allow a developer to quickly create components that move other GUI components between each other via drag and drop. This is particular useful for situations where a user can organize information between two or more lists by dragging a component that represents that information around on the screen. These classes are meant to decrease development time for DnD interfaces that are used solely within a single GUI. They are not meant to speed up dragging and dropping objects from a Java application to your OS's desktop or another running application, for instance.

A couple of LayoutManagers classes have also been created to help position components in a GUI. Java is notorious for lacking a set of LayoutManagers (or other GUI-based tools) that are easy to use, simple, and powerful. At best, only the first two of those options is supported at the same time. Java's Grid Bag layout manager is powerful, but difficult to use and requires code that isn't exactly pleasing to the eyes or easy to understand and/or maintain.

The first LayoutManager that was created to address some of the issues with Java's Grid Bag layout manager and is called LayeredLayout. It is called "layered" because the idea is that Components are layered on top of each other, with the top most component stretching to take up any remaining space because it "shows" through as the top layer.

To put components into a LayeredLayout either a "plot" must be specified for all of the components or a set of coordinates must be specified for each component. The plot or coordinates break up the layout area into a set of rows and columns. The dimensions of the rows and columns are determined by the layer they fall into. As row and column numbers increase, so do the layers. This way, the top-left most components are expected to only stretch until they reach other components to the right or below them, and the bottom-left most component will stretch to the edge of the container. A large percentage of layouts take on this look, so this layout can be used to easily create many GUIs.

The second LayoutManager, PositionLayout, was made with a more traditional position placement in mind. For a container that uses a PositionLayout, an initial coordinate "workspace" is set. This workspace is specified by a width and a height that all of the child components will base their positions off of. Each child component's position, size, and anchoring are specified. As the parent component is resized or moved, each child component will adjust according to these attributes. This kind of layout also lends itself well to reading in the positions of components from a tool that is used to place components via a GUI. It is very similar to how components are placed in a .NET GUI application.

Wizard Creator

Included in DB core's gui package is a package for creating custom GUI-based wizards. A Wizard is defined as having a particular WizardTask and a set of WizardPages that are used to accomplish that task. The WizardPages are kept in a WizardPagePool that can be navigated over via a WizardPageNavigator. A Wizard can either be imbedded in a WizardFrame or displayed inside of another Swing Component. Other features include a DynamicValidationWizardPageView that provides a means by which data can be dynamically validated against some algorithm while the user is entering it. This allows for rapid and satisfying user feedback.

The wizard package allows for highly-customizable GUI wizards to be quickly created.

Dynamic Signed Jar Class Loader

DB core's signed Jar/Class loader is used to load Java Jar files that have been digitally signed. It is different from Java's standard loader because it can be configured to refuse to load Jar files that haven't been signed by a particular entity. Java's loader will only throw an exception if a Jar file is not signed properly, not if the Jar file hasn't been signed at all (or has had its signature information stripped).

A Java keystore is associated with the loader and a set of aliases can be set that must have certificates that verify the signatures of the jar files or else they will fail to load. This design adds another level of security to digitally signed jar files -- disallowing unauthorized content from even being loaded.

Logging Framework

DB core's logging package provides a simple logging framework for easily creating individual Loggers and logging information based on different verbosity levels. The Loggers are thread-safe and customizable. The Loggers can be accessed globally via a LoggerManager class. Logging to the console or to a file is supported (includes log file rotation support). Log file recovery is also supported -- this means that if a log file is deleted during a log, the Logger will recover and create a new log file, it will not continue to write to a file descriptor that is, effectively, no longer valid.

Digital Bazaar Network Engine

DB core's net package contains a framework for creating versatile network applications. It is built upon a set of classes that abstract away lower-level network details and provide simple interfaces for communication over the web. It contains a fast, lightweight, and highly customizable "Micro Web Server" and support for HTTP and basic support for SOAP (Simple Object Access Protocol).

One of the core classes in the net package is WebConnection. This class is an interface for a connection over the web. The interface allows a developer to read and write from the connection, throttle bandwidth, disconnect the connection, and obtain information about the connection.

The read functionality of a WebConnection has 3 flavors. There is a regular "read" which will do an unblocked read from the connection. There is a "blockedRead" which will not stopping reading until a certain amount of data has been acquired or until the read is forcefully interrupted (which can be recovered from). And, lastly, there is a "unread" function, that allows a developer to "put back" a chunk of data that has been read from the connection so that it can be read again later. This function is particularly useful when some of the information sent over the connection can be used to determine who should be handling the data without ruining the data or having to pass it manually to the chosen data handler. This way, a data handler doesn't need any special implementation -- it merely reads from the connection as if no one had messed with it previously.

A WebConnection can be assigned BandwidthThrottlers to throttle how quickly one can read to or write from the connection. The BandwidthThrottler class uses an algorithm involving two-shifting time windows that measure the amount of data passing through a connection. This allows for an accurate limiting of bandwidth according to a particular desired averate rate. BandwidthThrottlers can also be shared across several different WebConnections so that a set of connections' total bandwidth is limited to a certain rate.

A basic TCP/IP implementation of the WebConnection interface is provided by GenericWebConnection.

A basic proxy TCP/IP implementation ProxyWebConnection extends GenericWebConnection and is used to proxy information from one WebConnection over another.

A WebConnectionClient is a client that can create WebConnections to a server and the WebRequest and WebResponse classes form abstract classes for sending requests and receiving responses over a WebConnection.

Micro Web Server

The WebConnectionServer class is the core class for DB's "Micro Web Server." It allows any number of WebConnectionHandlers to be attached to it that handle various different kinds of WebConnections. An AbstractWebConnectionHandler provides the basic implementation for all WebConnectionHandlers, such as the GenericWebConnectionHandler, and the SslWebConnectionHandler. Each WebConnectionHandler is assigned to a port on the server.

The server manages all of the various ports for its handlers and all of the incoming WebConnections via a WebConnectionAcceptor. Its WebConnectionAcceptor makes use of a thread pool for delegating threads for each WebConnection. Its thread pool is smart and will collect idle threads after a long enough period of time, which can be configured as desired. The number of connections the server will accept can also be configured.

Each WebConnectionHandler makes use of a WebConnectionServicer. The WebConnectionServicer actually services a WebConnection once it has been handled in whatever manner is appropriate. A SecureWebConnectionServicer can be used to run customized security checks on the WebConnection before it is serviced if so desired.

WebConnections can be terminated and disconnected by the server at any time, allowing for a clean shutdown or a forced shutdown.

A particular WebConnectionServicer is worthy of note: the ProxyPortWebConnectionServicer. The ProxyPortWebConnectionServicer is used to create a proxy between WebConnections over one external port and many internal ports. This servicer has its own internal WebConnectionServer that uses its own set of WebConnectionHandlers. When WebConnections are serviced by the ProxyPortWebConnectionServicer, they are analyzed by each of its WebConnectionHandlers to determine what kind of connection it is. Based on the connection type, it is passed off to the internal WebConnectionServer for handling. This design allows a developer to set up one piece of software that has only one exposed external port on its firewall that can receive a number of different protocols. For instance, a single port can be used for both SSL and non-SSL traffic. Furthermore the internal web server makes use of ephemeral ports so that there is no need to worry about configuring which ports it must open.

The ProxyPortWebConnectionServer provides the basic implementation for setting up a WebConnectionServer with a ProxyWebConnectionServicer and an internal WebConnectionServer.

HTTP Extensions to Micro Web Server

HttpWebRequest and HttpWebResponse implement basic HTTP requests and responses and extend the abstract classes WebRequest and WebResponse, respectively. These classes are quite flexible allowing the user to specify whatever HTTP headers they desire as well as allowing the user to transmit data easily in binary or text format. Multipart support is provided.

HttpWebConnection implements WebConnection and allows a user to easily send and receive http headers and bodies. Http content encoders/decoders and Http transfer encoders/decoders can be attached to handle special encodings of the data being transmitted and the transmission method, respectively. Two such encodings that are already implemented are gzip content encoding and HTTP/1.1 "chunked" transfer encoding.

The HttpWebConnectionServer adds HTTP support to a WebConnectionServer. The HttpProxyPortWebConnectionServer adds HTTP support to a ProxyPortWebConnectionServer.

Either of these HTTP servers allows HttpWebConnectionServicers to be attached to it. An HttpWebConnectionServicer reads an HttpWebRequest from an HttpWebConnection and prepares it for servicing.

HttpWebConnectionServicers use HttpWebRequestServicers to actually service HttpWebRequests and create HttpWebResponses.

Each one of these HttpWebRequestServicers can be assigned to a different "path." In this manner, many different kinds of HTTP-based services can be associated with a single server. For instance, if you had an HTTP service that provided a SOAP web service for getting information about dogs, you could attach an HttpWebConnectionServicer under the path "/dogs/". Another similar service could be listed under the path "/cats/". Another servicer should be used to simply provide HTTP GET access to a directory structure under the path "/myfiles/". An HttpGetRequestServicer has been provided to aid in the rapid development of a simple GET server.

WSDL Extensions to Micro Web Server

WSDL (Web Service Definition Language) is in extensive use with Web Services to provide a definition for how to use a particular Web Service. WSDL provides the definition for the types of messages that a Web Service supports so that a Web Client can make use of that service.

DB core's net.wsdl package provides classes for parsing WSDL's as well as generating them for Web Services. A developer that creates an interface file for a SOAP Web Service can use DB's wsdl extensions to generate a WSDL for its clients to make use of. DB's SoapClient can download a Web Service's WSDL and make use of it to make SOAP calls.

SOAP Extensions to Micro Web Server

Basic support for SOAP (Simple Object Access Protocol) is amongst the extensions for DB's Micro Web Server. To create a customized Soap Web Service using DB's technology, all that is needed is an interface file that defines the web operations that are to be exposed for clients to use, an implementation for that interface, and a class that extends AbstractSoapWebService to combine those two elements together and generate a WSDL.

The simplest example of this would be a Java interface file with one method in it that takes a String as a parameter and returns a String. Then a Java class file that implements that interface and returns the passed String in the method would be needed. Then a class that extends AbstractSoapWebService and passes its web service name, namespace, and the name of the Java interface class to the AbstractSoapWebService's constructor would be needed. This can be accomplished in under 10 minutes -- though your method wouldn't be particularly useful :).

The WSDL for the developer's service would be automatically created and none of the SOAP details would need to be seen.

ComplexTypes are not currently supported by DB's SOAP library. It is currently being debated as to whether or not they will be supported given the current direction of Web Services and XML. DB's technology makes extensive use of XML serialization to transport its objects, as do many other web applications. SOAP is seen as somewhat *not* simple for several reasons, one of which includes adding another way to serialize objects when transporting an XML string of the object would have sufficed. For now, we use XML to represent many of the objects in our systems, and so we simply transport them in XML format over SOAP.

Every Java primitive type is supported by DB's SOAP library. So method signatures that use Strings, longs, ints, doubles, floats, bytes, chars, booleans, etc -- are compatible with SOAP and will transfer nicely. We recommend converting Objects to XML and using Strings as parameters and return types for SOAP methods.

Live Data Stream Processing

DB core's stream package provides utilities for inspecting and modifying data as it is streamed.

The core classes and interfaces responsible for providing this functionality are ManagedInputStream, IStreamManager, and IStreamProcessor. These classes work in conjunction to allow an application to read data from a stream (ManagedInputStream), organize it into logical parts (IStreamManager), and process the data in some fashion (IStreamProcessor) before it is read by another stream.

Any number of ManagedInputStreams with different IStreamManagers and IStreamProcessors may be linked together to create powerful and customized live stream modifications.

Also included in DB core's stream package are tools for performing live stream encryption/decryption and compression/decompression.

Operation Engine

DB core's ope (Operation Engine) package provides an engine for executing arbitrary operations. It is a sort of application level OS that gives a developer more control over every operation that is being executed in his or her application.

Any Operation that can be executed on the engine must have a defined OperationExecutionEnvironment that is used to determine the conditions underwhich an Operation can be run. For instance, some Operations can only be run one at a time, or some number of a certain type of Operation may only be able to run concurrently. Also, some Operation may only be able to run if the OperationEngine is in a particular state (as set by other Operations).

The developer is provided with a great deal of control over the Operations that are being executed on an OperationEngine, including the ability to interrupt and/or terminate any executing or queued Operation.

A configurable thread pool is used to delegate threads for executing Operations, and Operations can be waited on for completion or run in a background process and execute a call back method when complete.

DB UPnP Client Stack

DB core's upnp library provides a basic implementation for the UPnP (Universal Plug and Play) protocol. It contains classes for implementing UPnP services and UPnP clients. The current implementation includes all of the basic client needs for UPnP communication -- so that any UPnP client can make use of any UPnP service, however, only some higher level interfaces have been written to abstract away the more specific details. That is, a developer could create their own UPnP client messages using DB core's upnp package, but they would have to provide a higher level abstract interface to wrap those messages for easier use.

The most commonly used UPnP client-side functionality has had such interfaces written for it. UPnP is used to edit a UPnP-capable router's firewall settings to forward ports. A very easy to use interface has been provided that allows a developer to open a port on a router's firewall and forward it to the appropriate destination.

To do this, a developer first instantiates an InternetGatewayControlPoint and calls discoverInternetGateways() to discover all of the available internet gateways. Then, the developer can retrieve a collection of discovered gateways (as InternetGatewayDeviceClients) with a call to getDiscoveredInternetGateways(). The gateways can be iterated over to either find the desired gateway via one of its properties, or to modify all of them. To create a port mapping, a developer instantiates a PortMapping object with the desired port and IP address settings, and then calls either addPortMapping() or overwritePortMapping() on the InternetGatewayDeviceClient object. To remove a port mapping, a call to removePortMapping() can be made.

XML Processing Utilities

DB core's xml package contains a number of XML tools for easing the development of XML serializable Objects. The most important of these tools is the XmlElement class. This class is effectively a wrapper for DOM elements and abstracts away some of the more tedious details of the DOM implementation (like having to drill down into text child nodes to get text from elements). It provides an easy to use interface for quickly converting Objects to XML and converting XML to Objects.

A class that is used at the core of DB's secure network communication is SignableXmlEnvelope. This class allows a string of XML (i.e. an XML-serialized Object) to be embedded in an envelope (or container) that can be digitally signed and verified. This allows for the secure transmission of complex data between two data points. This method can be used to ensure that the XML-serialized Object being transmitted has not been tampered with and that it was sent from an authorized entity.

Code has started to be written that makes use of Java 1.5's annotations -- to allow for extremely easy XML serialization of Objects (by simply marking data elements with annotations), however, it is currently incomplete.