Skip to content

Commit 77ebec6

Browse files
robinhood-jimrobinhood-jim
authored
Merge pull request #191 from robinhood-jim/develop
Develop
2 parents 5ab6151 + b7d6711 commit 77ebec6

34 files changed

+771
-139
lines changed

‎README.md‎

Lines changed: 73 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -8,40 +8,85 @@ Slightly Framework design to support Spring based java or Bigdata program.
88

99
1.Introduction
1010

11-
I.This project is base on Spring Framework and has four modules:
12-
|----------------------------------------------------------------------------------------------------|
13-
| Module | Description |
14-
|----------------------------------------------------------------------------------------------------|
15-
| Core | the core class include data access layer basic class(model,dao,service) and etc. |
16-
|----------------------------------------------------------------------------------------------------|
17-
| Comm | FileSystem Access tool(local/vfs),support FileFormat(csv/xml/json/avro/parquet/protobuf)|
18-
| | ,support Compress Format(gzip/bzip2/snappy/lzo/zip/lzma/lz4) |
19-
| | ,read and write excel,read word or PowerPoint |
20-
|----------------------------------------------------------------------------------------------------|
21-
|Hadooptool|FileSystem Access tool(hdfs), comm tool to access to HDFS,Hbase,Hive,Mongdb and etc |
22-
|----------------------------------------------------------------------------------------------------|
23-
|Example |springmvc config based and spring boot based Example; |
24-
|----------------------------------------------------------------------------------------------------|
25-
|Web |struts1,struts2 and springmvc support web component and required class. |
26-
|----------------------------------------------------------------------------------------------------|
27-
|Webui |Spring Boot with Oauth2 Thymeleaf Example; |
28-
|----------------------------------------------------------------------------------------------------|
29-
|Estool | ElasticSearch Comm Query tool |
30-
|----------------------------------------------------------------------------------------------------|
31-
|Tracer | Zipkin Brave tracing，Can trace All Database and Record parameters |
32-
|----------------------------------------------------------------------------------------------------|
33-
34-
II. Special feature
11+
I.This project is base on Spring Framework and has below modules:
12+
|----------------------------------------------------------------------------------------------------------------|
13+
| Module | Description |
14+
|----------------------------------------------------------------------------------------------------------------|
15+
| Core | the core class include data access layer basic class(model,dao,service) and etc. |
16+
|----------------------------------------------------------------------------------------------------------------|
17+
| Comm | FileSystem Access tool(local/vfs),support FileFormat(csv/xml/json/avro/parquet/protobuf)|
18+
| | ,support Compress Format(gzip/bzip2/snappy/lzo/zip/lzma/lz4) |
19+
| | ,read and write excel,read word or PowerPoint |
20+
|----------------------------------------------------------------------------------------------------------------|
21+
|Hadooptool |FileSystem Access tool(hdfs), comm tool to access to HDFS,Hbase,Hive,Mongdb and etc |
22+
|----------------------------------------------------------------------------------------------------------------|
23+
|Example |springmvc config based and spring boot based Simple framework Example; |
24+
|----------------------------------------------------------------------------------------------------------------|
25+
|Web |struts1,struts2 and springmvc support web component and required class. |
26+
|----------------------------------------------------------------------------------------------------------------|
27+
|Webui |Spring Boot with Oauth2 Thymeleaf Example; |
28+
|----------------------------------------------------------------------------------------------------------------|
29+
|Estool | ElasticSearch Comm Query tool |
30+
|----------------------------------------------------------------------------------------------------------------|
31+
|Tracer | Zipkin Brave tracing，Can trace All Database and Record parameters |
32+
|----------------------------------------------------------------------------------------------------------------|
33+
34+
35+
It is available under the terms of either the Apache Software License 2.0 or the Eclipse Public License 1.0.
36+
37+
2.Support Features
38+
I. Construct Simple Java FrameWork
39+
contain use sysrole and relation with customer privilege to use and roles;
40+
1.xml based standard frame : see example/config-example
41+
2.spring based standard frame: see example/boot-example
42+
43+
II. Bigdata supprot
44+
hadooptool:
45+
HDFS tool: com.robin.hadoop.hdfs can access HDFS with kerberos security
46+
Hbase tool: com.robin.hadoop.hbase hbase tool
47+
Cassandra tool : CassandraUtils
48+
49+
III. BigData common file format read/write tools support(including compress type support)
50+
AVRO
51+
PARQUET
52+
ORC
53+
PROTOBUF
54+
CSV
55+
XML
56+
JSON
57+
ARFF(weka format)
58+
59+
IV. File storage and Cloud storage support
60+
LocalFileSystem
61+
ApacheVFS
62+
HDFS
63+
Amazon S3
64+
Aliyun OSS
65+
Tencent COS
66+
Apache Kafka
67+
RabbitMq
68+
69+
V. Iterable and wirtable support intergate Storage and file format
70+
mix storage and File format to support cross storage read/write
71+
72+
VI. Spring cloud support
73+
WebUI simple webui base on dhtmlxGrid 5.1 with spring boot native
74+
related project in my another project microservices
75+
76+
VII. Zipkin Intergation
77+
trace sub project aimed to support All database to be tracable and can record query parameters.
78+
79+
VIII. Special feature
3580
a.A user defined xml Query config system,similar to mybatis,but easy config.
3681
b.Support defined annotation or jpa annotation in JdbcDao with ORM.
3782
c. BaseAnnotationService can access DB with minimize code,and use transaction with annotation.
3883
d.A common db access meta and util,can access all kind of db.
3984
e.Spring cloud based WebUI
4085
f.support Hadoop plateform
41-
42-
It is available under the terms of either the Apache Software License 2.0 or the Eclipse Public License 1.0.
43-
44-
2.Development
86+
g. Excel read write utils support auto merge columns and customer header define.
87+
88+
89+
3.Development
4590

4691
I.Model Layer:Simple ORM tool, Support JAVA JPA or my BaseObject Annotation
4792
Demostration：(support Composite primary key)
@@ -221,14 +266,5 @@ Slightly Framework design to support Spring based java or Bigdata program.
221266
222267
upon feature aim to simplify the work to develop standard MVC java code.
223268
224-
II. Bigdata supprot
225-
hadooptool:
226-
HDFS tool: com.robin.hadoop.hdfs can access HDFS with kerberos security
227-
Hbase tool: com.robin.hadoop.hbase hbase tool
228-
Cassandra tool : CassandraUtils
229-
230-
III. Spring cloud support
231-
WebUI simple webui base on dhtmlxGrid 5.1 with spring boot native
232-
related project in my another project microservices
233269

234270

‎common/src/main/java/com/robin/comm/util/xls/ExcelProcessor.java‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -232,8 +232,7 @@ public static Object readValue(Cell cell, String type, DateTimeFormatter format,
232232
break;
233233
case Const.META_TYPE_BIGINT:
234234
if (!StringUtils.isEmpty(str)) {
235-
Double d = Double.valueOf(str);
236-
strCell = d.longValue();
235+
strCell = Double.valueOf(str).longValue();
237236
} else {
238237
strCell = 0L;
239238
}

‎common/src/main/java/com/robin/core/fileaccess/iterator/AbstractFileIterator.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,4 +144,8 @@ public String getIdentifier() {
144144
public AbstractFileSystemAccessor getFileSystemAccessor(){
145145
return accessUtil;
146146
}
147+
148+
public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
149+
this.accessUtil = accessUtil;
150+
}
147151
}

‎common/src/main/java/com/robin/core/fileaccess/iterator/AbstractResIterator.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package com.robin.core.fileaccess.iterator;
22

33
import com.robin.core.base.exception.OperationNotSupportException;
4+
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
45
import com.robin.core.fileaccess.meta.DataCollectionMeta;
56
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
67
import org.slf4j.Logger;
@@ -51,4 +52,9 @@ public void setReader(BufferedReader reader) {
5152
public void setInputStream(InputStream inputStream) {
5253
throw new OperationNotSupportException("");
5354
}
55+
56+
@Override
57+
public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
58+
throw new OperationNotSupportException("");
59+
}
5460
}

‎common/src/main/java/com/robin/core/fileaccess/iterator/ArffFileIterator.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import com.google.common.collect.Lists;
44
import com.robin.core.base.util.Const;
5+
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
56
import com.robin.core.fileaccess.meta.DataCollectionMeta;
67
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
78
import org.springframework.util.CollectionUtils;

‎common/src/main/java/com/robin/core/fileaccess/iterator/IResourceIterator.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
package com.robin.core.fileaccess.iterator;
2+
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
3+
24
import java.io.BufferedReader;
35
import java.io.Closeable;
46
import java.io.InputStream;
@@ -11,4 +13,5 @@ public interface IResourceIterator extends Iterator<Map<String,Object>>, Closeab
1113
String getIdentifier();
1214
void setInputStream(InputStream inputStream);
1315
void setReader(BufferedReader reader);
16+
void setAccessUtil(AbstractFileSystemAccessor accessUtil);
1417
}

‎common/src/main/java/com/robin/core/fileaccess/iterator/JsonFileIterator.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
import com.google.gson.stream.JsonReader;
1919
import com.robin.core.base.util.Const;
2020
import com.robin.core.convert.util.ConvertUtil;
21+
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
2122
import com.robin.core.fileaccess.meta.DataCollectionMeta;
2223
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
2324
import org.springframework.util.CollectionUtils;

‎common/src/main/java/com/robin/core/fileaccess/iterator/PlainTextFileIterator.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
import com.robin.core.base.util.Const;
1919
import com.robin.core.base.util.StringUtils;
2020
import com.robin.core.convert.util.ConvertUtil;
21+
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
2122
import com.robin.core.fileaccess.meta.DataCollectionMeta;
2223
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
2324
import org.springframework.util.ObjectUtils;

‎common/src/main/java/com/robin/core/fileaccess/iterator/TextFileIteratorFactory.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,8 @@ public static AbstractFileIterator getProcessReaderIterator(DataCollectionMeta c
4949
Class<? extends IResourceIterator> iterclass=fileIterMap.get(fileType);
5050
try {
5151
if (!ObjectUtils.isEmpty(iterclass)) {
52-
iterator = (AbstractFileIterator) iterclass.getConstructor(DataCollectionMeta.class, AbstractFileSystemAccessor.class).newInstance(colmeta,utils);
52+
iterator = (AbstractFileIterator) iterclass.getConstructor(DataCollectionMeta.class).newInstance(colmeta);
53+
iterator.setAccessUtil(utils);
5354
}
5455
iterator.beforeProcess();
5556
}catch (Exception ex){

‎common/src/main/java/com/robin/core/fileaccess/iterator/XmlFileIterator.java‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
import com.robin.core.base.util.Const;
1919
import com.robin.core.convert.util.ConvertUtil;
20+
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
2021
import com.robin.core.fileaccess.meta.DataCollectionMeta;
2122
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
2223

@@ -44,7 +45,10 @@ public XmlFileIterator(){
4445
public XmlFileIterator(DataCollectionMeta metaList) {
4546
super(metaList);
4647
identifier= Const.FILEFORMATSTR.XML.getValue();
47-
48+
}
49+
public XmlFileIterator(DataCollectionMeta metaList, AbstractFileSystemAccessor accessor) {
50+
super(metaList,accessor);
51+
identifier= Const.FILEFORMATSTR.XML.getValue();
4852
}
4953
@Override
5054
public void beforeProcess() {

0 commit comments

Comments
 (0)