Java on ZRJ | 学习笔记

一个神奇的 spark 报错

Tue, 20 Nov 2018 22:01:59 +0800

写 spark 跑的时候遇到一个神奇的错误，报错如下

18/11/20 16:44:44 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:160)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:140)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:655)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:208)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:113)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)

在 spark web ui 上的报错就更加无厘头，说

Job aborted due to stage failure: Task 194 in stage 31.0 failed 4 times, most recent failure: Lost task 194.3 in stage 31.0 (TID 24324, 10.56.83.212, executor 73): UnknownReason

重新审视了一下代码，并没有发现问题，于是采用二分大法，加上日志看，同时也放狗搜，本以为这种虚无缥缈的报错是查不到啥的，结果居然有，看到这里， https://stackoverflow.com/questions/41338617/spark-could-not-find-coarsegrainedscheduler 说

Yeah now I know the meaning of that cryptic exception, the executor got killed because it exceeds the container threshold. There are couple of reasons that could happen but the first culprit is to check your job or try adding more nodes/executors to your cluster.

这里 https://blog.csdn.net/u013709270/article/details/78879869 也说

分析解决方案：

1、这个可能是一个资源问题，应该给任务分配更多的 cores 和Executors，并且分配更多的内存。并且需要给RDD分配更多的分区 2、在配置资源中加入这句话也许能解决你的问题：–conf spark.dynamicAllocation.enabled=false ——————— 作者：JeemyJohn 来源：CSDN 原文：https://blog.csdn.net/u013709270/article/details/78879869 版权声明：本文为博主原创文章，转载请附上博文链接！

好吧，先试试加大 executor 和内存看看，其实原来的 executor 已经够多了，200 个，但是 core 是 2，内存也是 2g，这回都搞到 4 看看

。。。

依然不行

加上打日志的逻辑，前面半段是能跑完的，看来，问题就出在后面半段了，其实这里的改动并不大，就是加了一个新的表的 rdd 的 join 逻辑而已

正在一筹莫展之际，回头仔细的翻了翻抛出来的堆栈，结果赫然发现一个 null pointer exception 在列，原来根本不是这次的改动引入的，而是之前的一个改动没有做空指针保护，人参啊。。

分割文件行数

Tue, 22 May 2018 17:32:57 +0800

可以将一个大文件分割成小文件，每个指定行数

package com.example;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;

public class SplitFile20180522 {

    public static void main(String[] args) throws IOException {
        String filePath = "D:\\downloads\\split\\20180521-3.csv";
        boolean skipFirstLine = true;
        int linePerFile = 10000000;
        System.out.println(filePath);
        try (BufferedReader br = new BufferedReader(new FileReader(filePath))) {
            String line;
            int lineCount = 0;
            int writeFileCount = 1;
            FileWriter fileWriter = null;
            if (skipFirstLine) {
                br.readLine();
            }
            while ((line = br.readLine()) != null) {
                lineCount++;
                if (fileWriter == null) {
                    fileWriter = new FileWriter(filePath + "_" + writeFileCount, false);
                }
                fileWriter.write(line + "\r\n");
                if (lineCount % linePerFile == 0) {
                    System.out.println("write file " + writeFileCount + " done");
                    fileWriter.close();
                    fileWriter = null;
                    writeFileCount++;
                }
            }
            if (fileWriter != null) {
                System.out.println("write file " + writeFileCount + " done");
                fileWriter.close();
            }
        }
    }

}

历史评论

house (2018-08-31 11:30:30):

咋还用上了java

ZRJ (2018-08-31 19:19:28):

啥顺手用啥，都搬砖那么多年了，还纠结语言问题嘛

Java 线程池试用

Sun, 11 Feb 2018 17:42:42 +0800

参考 http://www.importnew.com/19011.html 照猫画虎试用了 java 的线程池

package me.zrj.test.test20170607;

import java.util.Date;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

class MyTask implements Runnable {
    private int taskNum;

    public MyTask(int num) {
        this.taskNum = num;
    }

    public void run() {
        System.out.println((new Date()) + " starting task " + taskNum);
        try {
            Thread.sleep(4000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println((new Date()) + " task " + taskNum + " finished");
    }
}

public class ThreadPoolTest20180211 {
    public static void main(String[] args) {
        ThreadPoolExecutor executor = new ThreadPoolExecutor(5, 10, 200,
                TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(5));
        for (int i = 0; i < 15; i++) {
            MyTask myTask = new MyTask(i);
            executor.execute(myTask);
            System.out.println((new Date()) + " executor.getPoolSize() = " + executor.getPoolSize()
                    + ", " + "executor.getQueue().size() = " + executor.getQueue().size()
                    + ", " + "executor.getCompletedTaskCount() = " + executor.getCompletedTaskCount());
        }
        executor.shutdown();
    }
}

输出如下：

Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 1, executor.getQueue().size() = 0, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 2, executor.getQueue().size() = 0, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 1
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 3, executor.getQueue().size() = 0, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 2
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 4, executor.getQueue().size() = 0, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 3
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 5, executor.getQueue().size() = 0, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 4
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 5, executor.getQueue().size() = 1, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 5, executor.getQueue().size() = 2, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 5, executor.getQueue().size() = 3, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 5, executor.getQueue().size() = 4, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 5, executor.getQueue().size() = 5, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 6, executor.getQueue().size() = 5, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 10
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 7, executor.getQueue().size() = 5, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 11
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 8, executor.getQueue().size() = 5, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 12
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 9, executor.getQueue().size() = 5, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 13
Sun Feb 11 17:27:47 GMT+08:00 2018 executor.getPoolSize() = 10, executor.getQueue().size() = 5, executor.getCompletedTaskCount() = 0
Sun Feb 11 17:27:47 GMT+08:00 2018 starting task 14
Sun Feb 11 17:27:51 GMT+08:00 2018 task 0 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 task 1 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 starting task 5
Sun Feb 11 17:27:51 GMT+08:00 2018 starting task 6
Sun Feb 11 17:27:51 GMT+08:00 2018 task 3 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 task 12 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 starting task 7
Sun Feb 11 17:27:51 GMT+08:00 2018 task 10 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 starting task 8
Sun Feb 11 17:27:51 GMT+08:00 2018 starting task 9
Sun Feb 11 17:27:51 GMT+08:00 2018 task 4 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 task 11 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 task 2 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 task 13 finished
Sun Feb 11 17:27:51 GMT+08:00 2018 task 14 finished
Sun Feb 11 17:27:55 GMT+08:00 2018 task 5 finished
Sun Feb 11 17:27:55 GMT+08:00 2018 task 6 finished
Sun Feb 11 17:27:55 GMT+08:00 2018 task 8 finished
Sun Feb 11 17:27:55 GMT+08:00 2018 task 9 finished
Sun Feb 11 17:27:55 GMT+08:00 2018 task 7 finished

另外参考这个 http://wiki.jikexueyuan.com/project/java-concurrency/executor.html 也写的不错

单例，多线程的一些验证

Wed, 07 Jun 2017 19:34:18 +0800

作为一个基础知识，java 下的一种单例实现是这样的

package me.zrj.test.test20170607;

public class Singleton {
    private Singleton() {}
    
    private static Singleton instance;
    
    public static Singleton getInstance() {
        if (instance == null) {
            instance = new Singleton();
        }
        return instance;
    }
}

这种单例的写法的明显问题就是线程安全，说都知道这么说，那么怎么来复现这个线程安全问题呢

一个最直接的写法就是这样

package me.zrj.test.test20170607;

public class MyThread implements Runnable {
    
    @Override
    public void run() {        
        System.out.println(Singleton.getInstance());
    }
    
}

package me.zrj.test.test20170607;

public class App 
{
    public static void main( String[] args ) throws InterruptedException
    {
        System.out.println( "Hello World!" );        
        
        new Thread(new MyThread()).start();
        new Thread(new MyThread()).start();        

    }
    
}

但是这个复现的验证代码本身也是能看出来是有问题的，就是 new Thread 的先后 start 并不能保证他们在“同一时间”开始跑，而如果他们先后开始跑的时刻相差比较大，那么就无法复现那个单例的实现的 bug 了

但是。。。实际上验证的时候发现，这种写法几乎每次都能复现单例实现的 bug，由此可见，相比于 new 一个对象的耗时来说，new 一个线程的耗时还是更短的

那如果我们把这个验证的代码写成这样子呢

package me.zrj.test.test20170607;

public class App 
{
    public static void main( String[] args ) throws InterruptedException
    {
        System.out.println( "Hello World!" );        
        
        new Thread(new MyThread()).start();
        Thread.sleep(100);
        new Thread(new MyThread()).start();        

    }
    
}

这样的话，就可以看到两次拿回来的单例都是指向同一个地址了

那么，如果在这种情况下，我们还想能够复现那个单例实现的 bug，应该怎么来做呢，或者说，怎么来实现一个 countdownlatch 呢

按照理解去写，可以写成这样

package me.zrj.test.test20170607;

public class MyThread implements Runnable {
    
    synchronized void print() {
        try {
            wait();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println(Thread.currentThread().getName() + "\t" + Singleton.getInstance());
    }
    
    synchronized void go() {
        notifyAll();
    }
    
    @Override
    public void run() {       
        print();        
    }
    
}

package me.zrj.test.test20170607;

import java.util.Date;

public class App 
{
    public static void main( String[] args ) throws InterruptedException
    {
        System.out.println( "Hello World!" );        
        
        MyThread myThread = new MyThread();
        
        new Thread(myThread).start();
        Thread.sleep(100);
        new Thread(myThread).start();

        myThread.go();
        
        while (true) {
            Thread.sleep(1000);
            System.out.println(new Date());
        }
    }
    
}

但是发现这种写法会出现一个问题，就是虽然 notifyAll() 了，但是依然只有一个线程打印出来东西

看到这里， https://www.zhihu.com/question/37601861 说 notifyAll() 只是让所有线程去抢锁，但是其实还是只有一个线程能跑 synchronized 中的代码

所以真正的 countdownlatch 应该是这么来写

package me.zrj.test.test20170607;

import java.util.concurrent.CountDownLatch;

public class MyThread implements Runnable {
    
    CountDownLatch countDownLatch;
    
    public MyThread(CountDownLatch countDownLatch) {
        this.countDownLatch = countDownLatch;
    }
    
    synchronized void print() {
        try {
            countDownLatch.await();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println(Thread.currentThread().getName() + " " + Singleton.getInstance());
    }
    
    @Override
    public void run() {       
        print();        
    }
    
}

package me.zrj.test.test20170607;

import java.util.Date;
import java.util.concurrent.CountDownLatch;

public class App 
{
    public static void main( String[] args ) throws InterruptedException
    {
        System.out.println( "Hello World!" );        
        
        CountDownLatch countDownLatch = new CountDownLatch(1);
        
        MyThread myThread = new MyThread(countDownLatch);
        
        new Thread(myThread).start();
        Thread.sleep(100);
        new Thread(myThread).start();

        countDownLatch.countDown();
        
        while (true) {
            Thread.sleep(1000);
            System.out.println(new Date());
        }
    }
    
}

自行编译 saiku 的一些填坑记录

Tue, 18 Apr 2017 14:02:08 +0800

起因是说 jackson 的代码存在一个远程任意代码执行漏洞， http://bobao.360.cn/news/detail/4124.html ，而 saiku 又用到了 jackson，用的是 2.5.1

因此我们需要自行编译一个 saiku 的 pentaho 插件，把 jackson 的版本升上去

在此之前，我们用的是从 saiku 的官方下载回来的预编译好的 saiku-pentaho 插件，版本是 saiku-plugin-p7-3.11-SNAPSHOT

saiku 的代码拉下来之后是这样的

直接导入所有子工程到 eclipse 报错，例如出现以下的 error， Artifact has not been packaged yet. When used on reactor artifact, copy should be executed after packaging: see MDEP-187. (org.apache.maven.plugins:maven-dependency-plugin:2.8:copy:copy:package) pom.xml /saiku-bi-platform-plugin-p5 line 31 Maven Build Problem

Google 之后看到这里， http://blog.csdn.net/ahau10/article/details/53258663 说这个其实是 eclipse 的 maven 插件的问题

于是忽略，直接 skip test 跑 maven clean package，报错， Could not transfer artifact org.saikuanalytics:saiku-webapp:war:3.13 from/to Analytical Labs Repo (http://repo.meteorite.bi/repository/alabs-release-local/): connect timed out

这个第一反应就是公司内网的问题，在 .m2\settings.xml 里面给配置了 proxy 之后重试，依然无效，于是尝试用 Proxifier ，但是神器 Proxifier 只能走 socket 4 或者 5 的代理，要么就是 https，没有 http 的代理，我试着强制配了一个，也还是不行（这里其实引申出来一个问题，就是我们并不知道 mvn 在 package 的时候，对于一个具体的 denpendency，到底是从公司内网的 repo 拉取的，还是上公网拉取的，以及走没走代理，这些应该都是在 mvn 的参数里面可以有类似 verbose 的开关的，不过赶时间，没细究下去）

不过回头去看，编译的输出

[INFO] Saiku Module Project ............................... SUCCESS [  0.103 s]
[INFO] saiku - core libraries ............................. SUCCESS [  0.338 s]
[INFO] saiku olap util .................................... SUCCESS [  2.294 s]
[INFO] saiku-services ..................................... SUCCESS [ 47.300 s]
[INFO] saiku - web ........................................ SUCCESS [ 12.694 s]
[INFO] saiku-ui ........................................... SUCCESS [  4.776 s]
[INFO] saiku - webapp ..................................... SKIPPED
[INFO] saiku-server ....................................... FAILURE [01:31 min]
[INFO] saiku biserver plugin .............................. SUCCESS [ 14.063 s]
[INFO] saiku biserver plugin .............................. SUCCESS [  2.527 s]
[INFO] saiku biserver plugin .............................. SUCCESS [  3.291 s]

可以发现其实大部分还是能够编译过的，只有 saiku-server 这个是失败了，既然 saiku biserver plugin 能够编译过（上面的 maven 输出中没有输出版本号，其实 3 个 saiku biserver plugin 分别是 plugin 的 p5 p6 p7 版本），我们用的是 p7，那么直接在 p7 的 pom.xml 中修改 jackson 版本的 dependency，一开始配置了 jackson 的 2.9.0pr1，但是这个版本还在 pr 阶段，虽然能够 maven package 通过，出来一个插件的 zip 包，但是不知道会不会有潜在的问题

后来发现帖子中说 2.7.10 也修复了这个漏洞， http://bobao.360.cn/news/detail/4124.html ，于是改用 2.7.10，dependency 如下

<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId>
  <version>2.7.10</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.jaxrs</groupId>
  <artifactId>jackson-jaxrs-json-provider</artifactId>
  <version>2.7.10</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-annotations</artifactId>
  <version>2.7.10</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-core</artifactId>
  <version>2.7.10</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.jaxrs</groupId>
  <artifactId>jackson-jaxrs-base</artifactId>
  <version>2.7.10</version>
</dependency>
<dependency>
  <groupId>com.fasterxml.jackson.module</groupId>
  <artifactId>jackson-module-jaxb-annotations</artifactId>
  <version>2.7.10</version>
</dependency>

然后尝试 mvn clean package，编译的时候可以发现他有类似这样的输出

[INFO] Downloading: http://repo.meteorite.bi/repository/alabs-release-local/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://repo.meteorite.bi/repository/alabs-snapshot-local/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://nexus.pentaho.org/content/repositories/proxied-3rd-party-releases/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://nexus.pentaho.org/content/repositories/pentaho-public-release-repos/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://nexus.pentaho.org/content/repositories/public-snapshots/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: https://maven.atlassian.com/content/repositories/atlassian-public/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://repository.pentaho.org/artifactory/repo/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://clojars.org/repo/com/fasterxml/jackson/core/jackson-core/2.7.10/jackson-core-2.7.10.pom
[INFO] Downloading: http://repo.meteorite.bi/repository/alabs-release-local/com/fasterxml/jackson/core/jackson-annotations/2.7.10/jackson-annotations-2.7.10.pom
[INFO] Downloading: http://repo.meteorite.bi/repository/alabs-snapshot-local/com/fasterxml/jackson/core/jackson-annotations/2.7.10/jackson-annotations-2.7.10.pom
[INFO] Downloading: http://nexus.pentaho.org/content/repositories/proxied-3rd-party-releases/com/fasterxml/jackson/core/jackson-annotations/2.7.10/jackson-annotations-2.7.10.pom

可以看到，即使是对于 jackson 这种公开的库，他也依然优先走了 repo.meteorite.bi 和 nexus.pentaho.org 的 maven repo 去拉取，这样就导致速度上比较慢，这个估计是在某个 pom 文件中配置的 repo 的优先次序导致的

导致编译直接报错，Could not resolve dependencies for project org.saikuanalytics:saiku-bi-platform-plugin-p7:jar:3.13: Failed to collect dependencies at com.fasterxml.jackson.core:jackson-core:jar:2.7.10: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-core:jar:2.7.10: Could not transfer artifact com.fasterxml.jackson.core:jackson-core:pom:2.7.10 from/to Analytical Labs Repo (http://repo.meteorite.bi/repository/alabs-release-local/): connect timed out -> [Help 1]

于是尝试把 saiku-tag-3.13\saiku-bi-platform-plugin-p7\pom.xml 中的 repositories 标签整个注释掉，依然失败，继续尝试把 saiku-tag-3.13\pom.xml 的 repositories 标签注释掉，这次倒是去公司内网的 maven 拉取了，但是报错拉取不到，于是排查了一下，发现 2.7.10 貌似还没上到 maven 仓库中

于是改回用 2.9.0pr1，同时把上面提到的注释重新打开

这次倒是可以打包成功过了，得到一个 saiku-plugin-p7-3.11.zip 的包，放到 \pentaho-server\pentaho-solutions\system 中，重启 pentaho 就可以了

但是发现 loading 一直转，进不去，打开 chrome 的控制台，发现错误如下

而官方预编译的包的前端输出如下

可以发现错误的根源就是那个 saiku.min.js 里面的抛出的异常，TabSet is not defined，跟到这个文件，在 \pentaho-server\pentaho-solutions\system\saiku\ui\saiku.min.js 发现自己打包出来的 saiku.min.js 文件只有 5k，而官方预编译的有 303k

跟踪可以发现，这个 saiku.min.js 来自 saiku-ui 工程，这个工程下面看到一个 gruntfile.js，可以说明他是用 grunt 来编译，grunt 是一个前端的编译工具，资料可以参考 http://javascript.ruanyifeng.com/tool/grunt.html 等文章，这个 grunt 里面坑比较多，估计暂时填不平，因此直接从官方的预编译包中拉出来一个 saiku.min.js 替换上去，就可以打开页面了，然后就是部署 saiku license 等例行工作了

至此，基本上就得到一个自行编译的带 jackson 2.9.0pr1 + 官方预置的 saiku.min.js 的混合包

-——————-

2017-4-18 15:26:05 追加

关于 maven verbose 输出的问题，可以参考这里， http://stackoverflow.com/questions/16898801/how-do-i-get-verbose-output-from-maven-in-the-eclipse-console

关于代理的问题，后来发现原因是公司的内网有两个代理，其中一个不行，另外一个专门给开发网使用的代理是可以工作的

spring mvc 的 controller 函数的参数

Wed, 17 Feb 2016 22:24:04 +0800

学习 spring mvc 的过程中，发现他的 controller 的函数的入参很是神奇，变化多端，捉摸不定，查了一下，这里有说：http://stackoverflow.com/questions/8372957/how-does-spring-mvc-controller-method-parameter-work

The general answer is “Spring magic”; however, “Supported handler method arguments and return types” in the MVC chapter of the Spring reference guide has the exact answers to your questions.

在这里 http://stackoverflow.com/questions/12696222/spring-mvc-controller-method-signatures 和这里 http://docs.spring.io/spring/docs/3.1.x/spring-framework-reference/html/mvc.html#mvc-ann-arguments 可以看到更详细的说明

netty 初窥

Fri, 01 May 2015 16:33:43 +0800

先说结论，总体感觉很好，比 cpp 上的 nio 框架好。

缘起是一个同学来问 java web 的问题，帮他调了一下，两年没怎么碰过 java web 的东西了，各种生疏，而且在调试的过程中，也再次感觉到 java 这个东西实在不适合做 web，一个是重，ssh 框架一上来就给人压得喘不过气的感觉，新增一个功能，要 spring struts 里面 xml 各种配，访问一下数据库，需要 hibernate 里面一层层的 dao，看着就累，就算不用 ssh，自己写 servlet，也是各种繁琐，而且还不方便调试，做个什么改动，还要重启一下 web 容器，tomcat 也不是什么好东西，本身也复杂，总而言之，在做 web 这个事情上，跟动态脚本语言一比，几乎没有任何优势，唯一的优势可能就是效率？我宁可不要这个效率，反正量上来了最终瓶颈都是卡在数据库这块，逻辑层再快也没用。

不过转念一想到 netty，这个 java 社区里面著名的 nio 框架，想着以前都是只闻其声未见其人，趁着 eclipse 装着，也就是尝试了一把，整体下来的感觉非常好，相比 cpp 上自己的一些 nio 框架和一些自己写的 nio 框架，主要体现在以下一些方面：

标准规范，相比起来，cpp 上 nio 框架，品种繁多，挑的人眼花缭乱，libevent libev libuv，连名字都长得差不多，但是里面各种概念各种思想各种实现又有差别，在选择成本上就大了很多，再者这些框架，例如 libevent 来说，虽说是个 cpp 框架，但是 c 的味道还是很重，就显得不够干净利落，java 就好多了，社区中基本就以 netty 为事实上的标准，不要烦心怎么选，闭着眼睛用就行了。
概念清晰，相比起来，libevent 之流的框架，概念绕来绕去，cpp 的 nio 框架里面，除了 muduo 算是概念比较清晰的以外，其他的，我觉得不太好上手理解，而对于业务来说，能快速上手，能投入开发，才是王道，我们用一个框架不是为了来学习他的内部概念，而是为了来上线业务的，（虽然说用到后期，理解其内部概念也很重要，不过这是后话了），相比之下，netty 的概念就很清晰，channel，handler 的封装非常明了，这点从样例的 echo server 的代码，甚至从日志中，都能看的出来，先来看日志

2015-5-1 15:53:35 io.netty.util.internal.PlatformDependent <clinit>
信息: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
2015-5-1 15:53:35 io.netty.handler.logging.LoggingHandler channelRegistered
信息: [id: 0x7d92c9d5] REGISTERED
2015-5-1 15:53:35 io.netty.handler.logging.LoggingHandler bind
信息: [id: 0x7d92c9d5] BIND(0.0.0.0/0.0.0.0:8007)
2015-5-1 15:53:35 io.netty.handler.logging.LoggingHandler channelActive
信息: [id: 0x7d92c9d5, /0.0.0.0:8007] ACTIVE
2015-5-1 15:53:37 io.netty.handler.logging.LoggingHandler logMessage
信息: [id: 0x7d92c9d5, /0.0.0.0:8007] RECEIVED: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007]
2015-5-1 15:53:37 io.netty.handler.logging.LoggingHandler channelRegistered
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] REGISTERED
2015-5-1 15:53:37 io.netty.handler.logging.LoggingHandler channelActive
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] ACTIVE
2015-5-1 15:53:38 io.netty.handler.logging.LoggingHandler logMessage
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] RECEIVED(4B)
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 31 32 33 0a                                     |123.            |
+--------+-------------------------------------------------+----------------+
2015-5-1 15:53:38 io.netty.handler.logging.LoggingHandler logMessage
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] WRITE(4B)
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 31 32 33 0a                                     |123.            |
+--------+-------------------------------------------------+----------------+
2015-5-1 15:53:38 io.netty.handler.logging.LoggingHandler flush
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] FLUSH
2015-5-1 15:53:39 io.netty.handler.logging.LoggingHandler exceptionCaught
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] EXCEPTION: java.io.IOException: 远程主机强迫关闭了一个现有的连接。
java.io.IOException: 远程主机强迫关闭了一个现有的连接。
	at sun.nio.ch.SocketDispatcher.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:25)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
	at sun.nio.ch.IOUtil.read(IOUtil.java:206)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
	at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:256)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:619)
2015-5-1 15:53:39 io.netty.handler.logging.LoggingHandler close
信息: [id: 0xb07295eb, /127.0.0.1:51027 => /127.0.0.1:8007] CLOSE()
2015-5-1 15:53:39 io.netty.handler.logging.LoggingHandler channelInactive
信息: [id: 0xb07295eb, /127.0.0.1:51027 :> /127.0.0.1:8007] INACTIVE
2015-5-1 15:53:39 io.netty.handler.logging.LoggingHandler channelUnregistered
信息: [id: 0xb07295eb, /127.0.0.1:51027 :> /127.0.0.1:8007] UNREGISTERED

这个是拿 nc 当客户端，做的一次连接，可以看到，TCP 连接的整个过程清晰明了，状态机一步步的转换清清楚楚，不看代码只看日志都能把流程理清

下面再来看对应的代码，官方的样例代码在 http://netty.io/4.0/xref/io/netty/example/echo/package-summary.html，但是其实这个代码写的并不好，对于一个担负着 helloworld 功能的 echo 服务来说，还给他搞什么 SSL 支持，实属画蛇添足，精简过后的代码，可以看这里：

package me.zrj;

import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelOption;
import io.netty.channel.ChannelPipeline;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioServerSocketChannel;
import io.netty.handler.logging.LogLevel;
import io.netty.handler.logging.LoggingHandler;

public final class EchoServer {
	static final int PORT = Integer.parseInt(System.getProperty("port", "8007"));
	
	public static void main(String[] args) throws Exception {		
		EventLoopGroup bossGroup = new NioEventLoopGroup();
		EventLoopGroup workerGroup = new NioEventLoopGroup();
		try {
			ServerBootstrap b = new ServerBootstrap();
			b.group(bossGroup,workerGroup)
			.channel(NioServerSocketChannel.class)
			//.option(ChannelOption.SO_BACKLOG, 100)
			.handler(new LoggingHandler(LogLevel.INFO))
			.childHandler(new ChannelInitializer<SocketChannel>() {
				@Override
				public void initChannel(SocketChannel ch) throws Exception {
					ChannelPipeline p = ch.pipeline();					
					p.addLast(new LoggingHandler(LogLevel.INFO));
					p.addLast(new EchoServerHanlder());
				}
			});
			ChannelFuture f = b.bind(PORT).sync();
			f.channel().closeFuture().sync();
		} finally {
			bossGroup.shutdownGracefully();
			workerGroup.shutdownGracefully();
		}
	}
}

package me.zrj;

import io.netty.channel.ChannelHandler.Sharable;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInboundHandlerAdapter;

@Sharable
public class EchoServerHanlder extends ChannelInboundHandlerAdapter {
	@Override
	public void channelRead(ChannelHandlerContext ctx, Object msg) {
		ctx.write(msg);
	}
	
	@Override
	public void channelReadComplete(ChannelHandlerContext ctx) {
		ctx.flush();
	}
	
	@Override
	public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
		//cause.printStackTrace();
		ctx.close();
	}
}

话说前面的那个 static final int PORT = Integer.parseInt(System.getProperty(“port”, “8007”)); 的写法也真是脱了裤子放屁多此一举，不说也罢了。

调试方便，这个得益于 java vm 的跨平台，确实是有好处，相比开发 cpp，由于 linux 终端下没有好用的 IDE，所以开发的时候都是在 windows 下拿 visual studio 来写，完事了还要搭一个自动的同步环境，及时的把 windows 上的改动自动推到 linux 编译机上，再 make，然后才能跑起来，过冒烟测试，没问题了，才能往 svn 上交，来回一折腾，确实不方便，相比之下，java 可以直接在 windows 上写完了，跑起来，看着日志输出，爱怎么调怎么调，小农经济自给自足，说道日志输出，这里又要说两点，一个是虽然说，在多线程并发的环境下，日志几乎是唯一的 debug 手段，但是，在 cpp 上，是想单步也没的单步，拿 gdb 去单步的话，各种命令背的晕，而且想看个 STL 结构也是望洋兴叹基本等同于痴心妄想，visual studio 倒是可以很方便的单步，还带 STL 内容各种展开，但是奈何远水救不了近火，还是歇菜，然后另外一点就是再次要吐槽，cpp 社区里面，连个标准的日志库都没，这点相比 java 简直就是被甩几条街，这个还不止体现在日志这一个环节上，cpp 上连一个 string 的 spilt 都要自己动手才能丰衣足食，对于业务开发来说，要的是开发效率，不造轮子，string spilt 之流，应该是属于基本中的基本了。顺着这个再扯到 profiling 工具，java 不愧是企业级的开发平台，各种周边配套，服务完善，profiling 之类的任务基本就是衣来伸手饭来张口，而 cpp 虽然也有一些所谓的覆盖度，所谓的 profiling，但是奈何年迈，很多都是在没有多进程多线程混用的环境下可以工作的好，要是进程线程关系一复杂，就傻眼了，这点其实也难怪工具，主要是 cpp 里面 tricky 的写法太多，什么静态语法检查的事情，做起来确实是头疼，eclipse 就不一样了，IDE 可以做到对于代码的理解几乎是实时的，哪个地方 death code，哪个 package 是 import 了没用，哪里的 exception 抛了没接，分分钟给你指出来，不用过正式编译环节，很多问题在开发的时候顺手就解决掉了
易于部署，打包 JAR，往生产环境上一丢就完事了，java -jar **.jar 命令就跑起来，这年代的 linux 发行版，基本都自带 JRE，方便程度不亚于原生 cpp
后期维护，这个主要是涉及到团队协作，由于 java 天生就是一个适合工业化协作生产的语言，所以后期接手维护起来其实真是很有优势，这一点不仅体现在细微处，包括变量命名，缩进，括号，之类的编程规范上，而且还体现在稍微宏观的例如一个包进来了，怎么 decode，怎么业务逻辑处理，怎么回包，怎么日志，怎么落地，这些东西由于都有标准化工具，所以基本上换个不同的人来写，也都是八九不离十，能到达到不同的人写同一份代码的境界的，实现了同一个世界同一个梦想，one world one dream，同一批码农同样的代码的梦想

JVM虚拟内存调整

Thu, 19 Jul 2012 19:13:07 +0800

这次真的是为了调一个环境花费两三天的时间，太纠结了，现在每天都要去听实习那个没什么意思的课程，只能每天中午午休的时候和晚上放学回来抓紧时间写一点，每次都是稍稍有点状态思路刚活跃时间就到了，又要走人

这次异常调到后来，感觉弯路走了很多，于是开始有意识的用记事本记录下来，最开始的时候放在 MyEclispe 自带的 tomcat 5.5 里面，可以启动的起来，但是由于 5.5 不支持使用注释语法，于是自己换了个 tomcat 7 ，于是报异常，异常是

信息: validateJarFile(D:DocumentscodeMyEclipse86.metadataapache-tomcat-7.0.29webappsNeddyTravelWEB-INFlibel-api.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/el/Expression.class
2012-7-17 22:02:29 org.apache.catalina.loader.WebappClassLoader validateJarFile
信息: validateJarFile(D:DocumentscodeMyEclipse86.metadataapache-tomcat-7.0.29webappsNeddyTravelWEB-INFlibservlet-api.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax rvlet/Servlet.class
2012-7-17 22:02:29 org.apache.tomcat.util.digester.Digester endElement
严重: End event threw exception
java.lang.NoSuchMethodException: org.apache.catalina.deploy.WebXml addFilter

而当时的 web.xml 是这样子的

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" id="WebApp_ID" version="3.0">
  <display-name>NeddyTravel</display-name>
  <welcome-file-list>
    <welcome-file>index.html</welcome-file>
    <welcome-file>index.htm</welcome-file>
    <welcome-file>index.jsp</welcome-file>
    <welcome-file>default.html</welcome-file>
    <welcome-file>default.htm</welcome-file>
    <welcome-file>default.jsp</welcome-file>
  </welcome-file-list>
  <context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>/WEB-INF/applicationContext-*.xml</param-value>
  </context-param>
  <listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
  </listener>
  <filter>
    <display-name>Post</display-name>
    <filter-name>Post</filter-name>
    <filter-class>travel.filter.Post</filter-class>
  </filter>
  <filter-mapping>
    <filter-name>Post</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>
</web-app>

感觉上应该是没有什么问题，出于慎重还是二分法注释掉了很多，最后基本上整个文件注释掉了，还是不行，异常变成这样

2012-7-19 18:00:30 org.apache.catalina.startup.HostConfig deployDirectories
严重: Error waiting for multi-thread deployment of directories to completehostConfig.deployWar=Deploying web application archive {0}
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1023)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:474)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1413)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:313)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401)
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1140)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:785)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.io.DataInputStream.readUTF(DataInputStream.java:644)
	at java.io.DataInputStream.readUTF(DataInputStream.java:547)
	at org.apache.tomcat.util.bcel.classfile.ConstantUtf8.<init>(ConstantUtf8.java:48)
	at org.apache.tomcat.util.bcel.classfile.Constant.readConstant(Constant.java:129)
	at org.apache.tomcat.util.bcel.classfile.ConstantPool.<init>(ConstantPool.java:60)
	at org.apache.tomcat.util.bcel.classfile.ClassParser.readConstantPool(ClassParser.java:209)
	at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:119)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2066)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1942)
	at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1908)
	at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1893)
	at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1296)
	at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:855)
	at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:346)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5173)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1100)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1618)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	... 5 more

报内存溢出，一开始认为 tomcat 默认的内存应该是够的，于是就没有往调内存方面想，后来实在折腾不过，回到这个地方来，开始思考是否真的是 tomcat 的内存不够，写了这个代码用来查看虚拟机内存，

Runtime runtime = Runtime.getRuntime();
out.println(runtime.totalMemory()/1024);
out.println(runtime.freeMemory()/1024);

发现内存初始只有 8M 左右，真的挺小，于是上网搜，根据方法，在 MyEclipse 的下配了 JVM 的内存，就好了

折腾了许久的问题，居然这么傻逼，真是无颜面对江东父老啊

Tomcat配置多域名虚拟主机

Fri, 13 Jul 2012 23:54:16 +0800

国内的 Java 社区资料确实丰富，可惜大都是抄来抄去，质量也是良莠不齐，比如这个简单的 tomcat 配置多余名虚拟主机问题吧，google 了一下，出来很多文章，前面的点进去看了看，发现解决不了问题，话都没有说清楚，于是关掉，点后面，结果坑爹的发现其实是抄前面的几篇，连错别字都不带改的，于是只能自己狂看好多篇然后挨个试，最后不容易终于试出来了，唉，吐槽完了，转正题。

首先，先明确一些简单的概念，访问一个 web 服务器的根本方法是使用 IP ，但是一般来说，一台主机的计算能力是有余的，于是可以同时运行多个域名下的网站，当然，出于另外的一些目的，我们也需要配置自定义域名，例如，tomcat 默认跑在 8080 ，而且一个网站是在子目录的形式访问的，也就是类似 http://localhost:8080/website/ 这样的 URL ，这样不利于我们在调试阶段使用基于根目录的相对路径形式，所以我的方法是在本地的时候配成 website.local 的形式，等到上线了就是 website.com 的形式，这样所有的相对路径都是可移植的。

主要配置的文件是 tomcat 目录下的 /conf/server.xml 文件，初始的文件结构是类似这样的

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 --><Server port="8005" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener SSLEngine="on" className="org.apache.catalina.core.AprLifecycleListener"/>
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener"/>
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"/>
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/>

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource auth="Container" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" name="UserDatabase" pathname="conf/tomcat-users.xml" type="org.apache.catalina.UserDatabase"/>
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    -->

    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector connectionTimeout="20000" port="80" protocol="HTTP/1.1" redirectPort="8443"/>
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"/>

    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine defaultHost="localhost" name="Catalina">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host appBase="webapps" autoDeploy="true" name="localhost" unpackWARs="true" xmlNamespaceAware="false" xmlValidation="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

我们需要做的就是在 Engine 标签下再增加一个 Host 标签，类似这样

<Host name="neddytravel.local" debug="0" appBase="D:DocumentscodeMyEclipse86.metadata.me_tcatwebappsNeddyTravel">
	<Context path="" docBase="." debug="0" />
</Host>

这里的 name 属性就是我们要自定义的域名， appBase 属性是这个网站的文件的真实路径，至于 debug 属性，一般都没看到有解释， tomcat 官网的解释是

The level of debugging detail logged by this Engine to the associatedLogger. Higher numbers generate more detailed output. If not specified, the default debugging detail level is zero (0).

然后就是下面的 Context 标签了，根据 http://www.wangchao.net.cn/bbsdetail_61162.html

path 表示在浏览器里的目录，空表示根目录, docBase 表示在硬盘上所在的目录

当然还要修改端口号为 80 ，修改完成后完整的文件是类似这样的

<?xml version="1.0" encoding="UTF-8" standalone="no"?><!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 --><Server port="8005" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener SSLEngine="on" className="org.apache.catalina.core.AprLifecycleListener"/>
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener"/>
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"/>
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/>

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource auth="Container" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" name="UserDatabase" pathname="conf/tomcat-users.xml" type="org.apache.catalina.UserDatabase"/>
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    -->


    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector connectionTimeout="20000" port="80" protocol="HTTP/1.1" redirectPort="8443"/>
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"/>


    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine defaultHost="localhost" name="Catalina">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>


      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host appBase="webapps" autoDeploy="true" name="localhost" unpackWARs="true" xmlNamespaceAware="false" xmlValidation="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>

	  <Host name="neddytravel.local" debug="0" appBase="D:DocumentscodeMyEclipse86.metadata.me_tcatwebappsNeddyTravel">
          <Context path="" docBase="." debug="0" />
	  </Host>

    </Engine>
  </Service>
</Server>

配置文件的修改到这里就结束了，下面修改 hosts 文件，至于不理解 hosts 文件的作用的话，请自行搜索，这方面资料很多，也讲的不错

hosts 文件在 C:WindowsSystem32driversetc ，打开，追加一行

127.0.0.1 neddytravel.local

就可以了，然后重启 tomcat ，在浏览器中访问自定义的域名 neddytravel.local 应该就可以看到页面了