When you’re using onejar 1.4.5 or above you want to turn off the noise the bootloader makes.
java -Done-jar.verbose=false -jar service.jar
Posts Tagged ‘Java’
0
Vertx 3 – FatJar alternative
Posted: May 15, 2015 in Java, MicroservicesTags: Java, Maven, Onejar, Shader, Vertx
Vertx3 is nearing its release date and we’re experimenting with the upcoming release to minimize future surprises. One of the major changes is the generation of fatjars. The Vertx2 fatJar plugin doesn’t exist anymore in the Vertx 3 world. Now Vertx 3 uses the maven-shade-plugin to generate the fat JAR. Unfortunately, the shader has its own problems. For example, it is very cumbersome to handle duplicate files especially in transitive dependencies which are something out of your control.
When you’re running into unsolvable errors with the maven-shade-plugin, you can easily switch to the onejar-maven-plugin. Configure your POM as follows:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<configuration>
<archive>
<manifestEntries>
<Main-Class>io.vertx.core.Starter</Main-Class>
<Main-Verticle>io.vertx.example.HelloWorldVerticle</Main-Verticle>
</manifestEntries>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.dstovall</groupId>
<artifactId>onejar-maven-plugin</artifactId>
<version>1.4.5</version>
<executions>
<execution>
<configuration>
<attachToBuild>true</attachToBuild>
<filename>service.jar</filename>
</configuration>
<goals>
<goal>one-jar</goal>
</goals>
</execution>
</executions>
</plugin>
This will create a new fat JAR with the project JARs as a complete file on the inside. They are not unpacked, so there are no issues with overwriting files in the META-INF. The downside is that they use an own boot-mechanism.
This again has some problems when for example certain JARs use the SPI mechanism or self-cooked mechanisms to load classes.
If you want to stream an incoming request, you can use the following code:
final Handler<HttpServerRequest> streamer = (HttpServerRequest request) -> {
HttpServerResponse response = request.response();
container.logger().info("Streaming... ");
long ts = System.currentTimeMillis();
// handle the content
request.dataHandler((Buffer data) -> {
container.logger().info("Received " + data.length());
});
request.endHandler(v -> {
container.logger().info("Done! " + (System.currentTimeMillis() - ts));
request.response().end();
});
// handle upload errors
request.exceptionHandler(
(Throwable throwable) -> {
throwable.printStackTrace();
response.setStatusCode(HttpResponseStatus.BAD_REQUEST.code()).end();
}
);
};
The current threading model in Vertx 2 is very powerful. So powerful that you can easily put yourself out of business. In a project I’m currently working on, we created one (micro)service which does no more than store data in a MongoDB instance and creates a in-memory Lucene to provide a search functionallity. The functionality was trivial. There is a REST interface to do basic CRUD operation, a search resource and one special feature: an importer.
The search-resource contains nothing more than code to publish a message to a Lucene worker-verticle. This worker-verticle uses a LocalHandler to listen for events. And all worked well, since we had more then +2000 wildcard-search-operation per seconds. Internally we have 20 worker threads.
As we went on with the development, we encountered a problem during the import-process last week. A legacy system uploads a file to the import resource of the service. The importer receives the file, validates the content, and starts the import. The data was first stored in the MongoDB and when this call was finished, the same data was added to the Lucene index across the cluster.
One problem however was that all the worker-threads were blocked during the import-process leaving no room for parallel requests. After a brief search we saw that MongoDB-verticle was consuming all the worker-threads leaving no threads available for the Lucene-Index-worker. Like this, the node is not reachable and we had the negative effect that the supervision-service was marking this node as being down. We had to reduce the number of threads of the MongoDB-verticle. But, looking in the GitHub code of this verticle, we saw that the module had the property “multithreaded”: true in the mod.json. This is sometimes good, but in our case bad. We’ve forked this project and made the verticle no longer multithreaded. To have enough capacity during the import, we’ve instantiated the MongoDB-verticle five times. We currently run with 20 worker threads, so we always have room to keep the search function alive.
Lessons learned is that even in predefined modules, you need to fine-tune the threading-strategy.
Versioning and Continuous Integration – Part 1
Posted: December 13, 2014 in Continuous Integration, MavenTags: CI, Java, Maven
When you want to do Continuous Integration, Delivery and Deployment, you sometimes want to use the latest available artifacts in your builds. Just as with all things in life, there are people who oppose and people who advocate this approach. Of course, this will not be the silver bullet, but for certain projects it can be useful. In those kind of projects you just want to reference a range of versions for a certain artifact instead of relying on a static version number. Let me illustrate this with an example.
Let’s consider the following scenario. Suppose you work on a web-project which depends on a smaller library. Let’s assume it is a simple, plain JAR file with tests etc. This JAR file is of course subject to frequent change. You can build your build-pipeline in such way that your JAR is compiled, tested and delivered on every change. When a new artifact is uploaded to the repository, you can automatically trigger a new build of your web-project.
When you use maven, the difficult question is how you set a plausible version number without checking the code back in (which would trigger a loop in your build-process). Then we must solve the question about how the web-project will reference the library. Most likely, your project is stuck on the dependency in version X.Y.Z. The version number is not increased on every build because it is not patched nor did the functionality change. Moreover, you don’t want to check out the web-project every time to increase the version number of the depending library manually.
A solution here is to enhance your versioning system with a build number and use ranges. Both needed features are supported by Maven itself (time of writing 3.2.3) and the versions-plugins.
The Version Problem
Maven is not that great when it comes to CI. The way Maven handles the versions is not very flexible. But, if you really think about the problem, you’ll realize that it is the way how software is versioned that seems to be problematic. The classical X.Y.Z scheme is particularly helpful for humans and human controlled workflows. The decision which number (major, minor, patch) has to be increased, remains a human decision. This has definitely advantages. Through this version-scheme, you communicate with the outside world how drastically your software has changes.
But it is very impractical for a CI system because it can’t automate this way of thinking. The CI only knows about the number of builds it has made. The CI just knows build #1, #2, etc etc.
You don’t want to loose the version-information nor the build number. A possible solution is to combine the two as one. You can change the versioning scheme to X.Y.Z-buildnumber.
The Library POM
Using the above scheme is nice. But, there is one trade-off. You don’t want to store the buildnumber in the source-code. This has been done in the past, as I can tell out of my own experience and to a certain extent this was plausible and functioned well. A drawback was that the Maven Release Plugin stored data in SVN itself.
Our goal is to keep the POM as clean and simple as possible. We don’t want to pollute the version-number either. The scheme X.Y.Z-buildnumber is useful in the world of CI but not in your local environment where the X.Y.Z scheme is the one which is under your control. So, let’s keep it simple and use the X.Y.Z scheme in the source-code and during development.
There is no need to have a build number locally nor in your source-code.
Building the Library
When the CI builds the library, it can add the build-number on the fly. This example uses the GO pipeline counter as the build number we want to add to the version-number. Unfortunately you cannot append it easily to the POM. You first have to read the version, add the build-number yourself and use it in the POM. Luckily we can use the power of Linux to put this in one line:
mvn versions:set -DnewVersion=$(mvn help:evaluate -Dexpression=project.version 2>/dev/null| grep -v "^\[")-$GO_PIPELINE_COUNTER
The version number is not checked in. Committing the code back to Git would trigger a new build and put us in a loop. We gave this a lot of thought and since there is no real use-case for this, we decided not to do it. It doesn’t solve nor create problems.
A note on placeholders
In earlier days it was possible to use placeholders in the version-tag. In Maven3 this is prohibited except for three fields. When you use the version-tag in the source-code, you locally have a problem because you have to provide the information which is cumbersome and error-prone. So, we decided not to use this system although you could easily set them using the CLI in your CI-environment.
The POM with the Dependency
In our example, we want to automatically build a dependent project when changes in the above library occurs. You have to be aware that this approach has advantages but could also have some disadvantages. You could easily break the software when there are non-backward-compatible changes. In our philosophy, that is a good thing. We need to keep our software up and running with the newest libraries and this involves break and incompatibilities. We need to detect failures as soon as possible. Of course, this is not the wanted behavior for all kinds of projects. You need to make the decision yourself if you want this system or not.
Let’s continue. In the depending project, you define a dependency but not with a static version but with a range:
<dependency> ... <version>[1.0.0, )</version> </dependency>
This defines the range of version you want to compile with. This is an open end version, so we will try to compile with every new version of the library. When the compilation or tests fail, we will be informed so we can fix the issue as soon as possible.
Normally seen, Maven should be able to resolve the ranges, but due to some bugs (currently we use Maven 3.2.3), the range do not gives the correct artifact. But there is a work-around with the versions-plugin. In a first step, you resolve the ranges:
mvn versions:resolve-ranges
This will replace the ranges with the latest version found in the repository. Your artifact will be built and tested with the latest dependency. Eventually you can auto-deploy your artifact directly.
Brain/Drain/Pain 