大数据maven（本地Java大数据环境基础配置）

创建项目

准备pom.xml文件配置（附在文档最后）

在下载jar过程中极其缓慢，简直不能忍受，查阅了下资料，得到下面的办法：

1、 maven仓库中没有支持cdh的相关依赖。cloudera自己建立了一个相关的仓库。要想利用maven添加相关依赖，则必须单独添加cloudera仓库

2、 在国内使用maven中央仓库一般会有网络问题，大部分人会使用aliyun仓库或者其他开源的仓库。所以需要修改setting.xml (以下配置中 *,!cloudera 表示除了aliyun仓库还使用cloudera仓库)

配置：

<mirror> <id>nexus-aliyun</id> <mirrorOf>*,!cloudera</mirrorOf> <name>Nexus aliyun</name> <url> http://maven.aliyun.com/nexus/content/groups/public </url> </mirror>

配置环境

下面两个配置表示是在本地运行还是到Linux环境中运行

dev

编写内容

local

编写内容

准备log4j的日志配置，这个网上很多，可以直接复制一份，改下关键地方就可以

HDFS连接地址的配置

hbase的连接配置

配置文件transformer-env.xml

文件内容

pom.xml文件配置

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.xlgl.wzy</groupId> <artifactId>localenv</artifactId> <version>0.0.1-SNAPSHOT</version> <properties> <hadoop.version>2.7.3</hadoop.version> <hbase.version>1.2.5</hbase.version> <hive.version>2.1.1</hive.version> <mysql.version>5.1.6</mysql.version> <gson.version>2.6.2</gson.version> <uasparser.version>0.6.1</uasparser.version> </properties> <repositories> <!-- 指定该项目可以从哪些地方下载依赖包 --> <repository> <id>aliyun</id> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> </repository> <repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository> <repository> <id>jboss</id> <url>http://repository.jboss.org/nexus/content/groups/public</url> </repository> </repositories> <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>${hadoop.version}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>${hbase.version}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> <version>${hbase.version}</version> </dependency> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>${hive.version}</version> </dependency> <!-- mysql start --> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>${mysql.version}</version> </dependency> <!-- mysql end --> <dependency> <groupId>cz.mallat.uasparser</groupId> <artifactId>uasparser</artifactId> <version>${uasparser.version}</version> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>${gson.version}</version> </dependency> </dependencies> <profiles> <!-- 指定环境 --> <profile> <!-- 指定本地运行环境，windows环境 --> <id>local</id> <activation> <activeByDefault>true</activeByDefault> </activation> <build> <resources> <resource> <directory>configuration/local</directory> </resource> <resource> <directory>src/main/resources</directory> </resource> </resources> <plugins> <plugin> <!-- 支持多个不同文件夹中的java代码进行编译 --> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>1.4</version> <executions> <execution> <id>add-source</id> <phase>generate-sources</phase> <goals> <goal>add-source</goal> </goals> <configuration> <sources> <source>${basedir}/src/main/java</source> <source>${basedir}/src/main/extr</source> </sources> </configuration> </execution> </executions> </plugin> </plugins> </build> </profile> <profile> <!-- hadoop环境运行， linux环境 --> <id>dev</id> <build> <resources> <resource> <directory>configuration/dev</directory> </resource> <resource> <directory>src/main/resources</directory> </resource> </resources> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>1.4</version> <executions> <execution> <id>add-source</id> <phase>generate-sources</phase> <goals> <goal>add-source</goal> </goals> <configuration> <sources> <source>${basedir}/src/main/java</source> </sources> </configuration> </execution> </executions> </plugin> <plugin> <!-- 该插件的作用是将jar文件打包到最终形成的项目jar文件中 --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.1</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <includes> <include>cz.mallat.uasparser:uasparser</include> <include>net.sourceforge.jregex:jregex</include> <include>mysql:mysql-connector-java</include> </includes> </artifactSet> </configuration> </execution> </executions> </plugin> </plugins> </build> </profile> </profiles> <build> <testSourceDirectory>src/test/java</testSourceDirectory> <plugins> <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>assembly</goal> </goals> </execution> </executions> </plugin> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.3</version> <configuration> <source>1.8</source> <target>1.8</target> <encoding>utf8</encoding> <!-- 默认是gbk --> </configuration> </plugin> </plugins> <pluginManagement> <plugins> <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.--> <plugin> <groupId>org.eclipse.m2e</groupId> <artifactId>lifecycle-mapping</artifactId> <version>1.0.0</version> <configuration> <lifecycleMappingMetadata> <pluginExecutions> <pluginExecution> <pluginExecutionFilter> <groupId>org.codehaus.mojo</groupId> <artifactId> build-helper-maven-plugin </artifactId> <versionRange>[1.4,)</versionRange> <goals> <goal>add-source</goal> </goals> </pluginExecutionFilter> <action> <ignore></ignore> </action> </pluginExecution> </pluginExecutions> </lifecycleMappingMetadata> </configuration> </plugin> </plugins> </pluginManagement> </build>