原文链接:点击打开链接
点击打开链接
点击打开链接
避免原文打不开,记录如下:
背景:
本人因为某Android项目需要,需要在Android中定义一个容量为2万的float数组。这两万个float值已经存在某个文件中。
方法:
1.直接在Java文件里定义?
Java单个函数的长度限制为65535字节(不确定具体数值,但肯定是有限制的)。故在Java文件中直接定义如此之长的数组是行不通的。
2. BufferedReader 或是 Scanner?
可以创建一个文件,每行存放一个float值。通过BufferedReader.nextLine()读取每一行的字符,用Float.parseFloat(String)来将读到的字符转换成float值,最后将float值存取至数组里。
也可以通过Scanner.nextFloat()这样的方法来读取文件中存放的float值。这个时候,文件里并不需要将每个float值放在单独的一行里。但是不同的float值至少需要以某些符号隔开(例如,空格)。
经过对比,通过BufferedReader读取2W个float值比较快,使用Scanner会占用大量内存,并且不断调用GC,导致线程暂停。但是,当要读取的文件很大的时候,即使使用BufferedReader也会出现问题:在低端的机子上运行时,Logcat会出现OutOfMemory的错误提示。
3. 将2万个float值写入文件后做Java的内存映射。
大家可以借鉴一下我测试对比所用的源码。第一个部分,我使用Scanner;第二个部分,我使用BufferedReader;第三部分,我将第二部分读取到的float值以byte形式存取至文件中;第四部分,我使用MappedByteBuffer将文件中的内容映射至内存,并生成新的float数组。
以下是结果运行截图:

当然,以上测试程序均在PC java端运行。真正要移植到Android上进行内存映射,还需要:
1. 由于从assets里读文件只能获取InputStream,而调用FileChannel.map()必须从FileInputStream中获得,故需要将assets里的内存映射文件转移至sd卡上后再从sd卡上读取它们。
2.在测试文件中,我使用了DataOutputStream.writeFloat()将float数组写入内存映射文件。该方法写入遵循BIG_ENDIAN规则。故当进行内存映射的时候,很有可能是以LITTLE_ENDIAN的方法读取的(这还要看你是如何实现内存映射的)。
以下这段代码,给出了如何将一个存好的float数组(cubeTextureCoordinateData)以LITTLE_ENDIAN的方式写入内存映射文件:
/* Writing Memory Map Raw Data */
ByteBuffer littleEndian = ByteBuffer.allocateDirect(11160 * 4).order(ByteOrder.LITTLE_ENDIAN);
i = 0;
while (i < 11160) {
littleEndian.putFloat(cubeTextureCoordinateData[i++]);
}
littleEndian.flip();
try {
FileOutputStream fos = new FileOutputStream("memorymap_texture_little_endian");
FileChannel fc = fos.getChannel();
fc.write(littleEndian);
fc.close();
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
Power of Java MemoryMapped File
In JDK 1.4 an interesting feature of Memory mapped file was added to Java, which allows to map any file to OS memory for efficient reading. A memory mapped file can be used to develop an IPC type
of solution. This article is an experiment with memory mapped file to create IPC.
Some details about Memory Mapped File, definition from WIKI
A memory-mapped file is a segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. This resource is typically a file that is physically present on-disk, but can also be a device,
shared memory object, or other resource that the operating system can reference through a file descriptor. Once present, this correlation between the file and the memory space permits applications to treat the mapped portion as if it were primary memory.
Sample Program
Below we have two Java programs, one is a writer and the other is a reader. The writer is the producer and tries to write to Memory Mapped file, the reader is the consumer and it reads messages from the memory mapped file. This is just a sample program to show
you the idea, it does’t handle many edge cases but it is good enough to build something on top of a memory mapped file.
MemoryMapWriter
02 |
import java.io.FileNotFoundException; |
03 |
import java.io.IOException; |
04 |
import java.io.RandomAccessFile; |
05 |
import java.nio.MappedByteBuffer; |
06 |
import java.nio.channels.FileChannel; |
08 |
public class MemoryMapWriter
{ |
10 |
public static void main(String[]
args) throws FileNotFoundException,
IOException, InterruptedException { |
11 |
File
f = new File( "c:/tmp/mapped.txt" ); |
14 |
FileChannel
fc = new RandomAccessFile(f, "rw" ).getChannel(); |
16 |
long bufferSize= 8 * 1000 ; |
17 |
MappedByteBuffer
mem =fc.map(FileChannel.MapMode.READ_WRITE, 0 ,
bufferSize); |
22 |
long startT
= System.currentTimeMillis(); |
23 |
long noOfMessage
= HUNDREDK * 10 * 10 ; |
26 |
if (!mem.hasRemaining()) |
28 |
start+=mem.position(); |
29 |
mem
=fc.map(FileChannel.MapMode.READ_WRITE, start, bufferSize); |
33 |
if (counter
> noOfMessage ) |
36 |
long endT
= System.currentTimeMillis(); |
37 |
long tot
= endT - startT; |
38 |
System.out.println(String.format( "No
Of Message %s , Time(ms) %s " ,noOfMessage,
tot)) ; |
MemoryMapReader
02 |
import java.io.FileNotFoundException; |
03 |
import java.io.IOException; |
04 |
import java.io.RandomAccessFile; |
05 |
import java.nio.MappedByteBuffer; |
06 |
import java.nio.channels.FileChannel; |
08 |
public class MemoryMapReader
{ |
13 |
*
@throws FileNotFoundException |
14 |
*
@throws InterruptedException |
16 |
public static void main(String[]
args) throws FileNotFoundException,
IOException, InterruptedException { |
18 |
FileChannel
fc = new RandomAccessFile( new File( "c:/tmp/mapped.txt" ), "rw" ).getChannel(); |
20 |
long bufferSize= 8 * 1000 ; |
21 |
MappedByteBuffer
mem = fc.map(FileChannel.MapMode.READ_ONLY, 0 ,
bufferSize); |
22 |
long oldSize=fc.size(); |
27 |
long startTime
= System.currentTimeMillis(); |
32 |
while (mem.hasRemaining()) |
34 |
lastValue=mem.getLong(); |
37 |
if (currentPos
< oldSize) |
40 |
xx
= xx + mem.position(); |
41 |
mem
= fc.map(FileChannel.MapMode.READ_ONLY,xx, bufferSize); |
46 |
long end
= System.currentTimeMillis(); |
47 |
long tot
= end-startTime; |
48 |
System.out.println(String.format( "Last
Value Read %s , Time(ms) %s " ,lastValue,
tot)); |
49 |
System.out.println( "Waiting
for message" ); |
52 |
long newSize=fc.size(); |
56 |
xx
= xx + mem.position(); |
57 |
mem
= fc.map(FileChannel.MapMode.READ_ONLY,xx , oldSize-xx); |
58 |
System.out.println( "Got
some data" ); |
Observation
Using a memory mapped file can be a very good option for developing Inter Process communication, throughput is also reasonably well for both produce & consumer. Performance stats by run producer and consumer together:
Each message is one long number
Produce – 10 Million message – 16(s)
Consumer – 10 Million message 0.6(s)
A very simple message is used to show you the idea, but it can be any type of complex message, but when there is complex data structure then serialization can add to overhead. There are many techniques to get over that overhead. More in next blog.
Java NIO - Channels and Memory mapping
- Channel- A channel represents a connection to a file, socket or any other component that can perform IO operations. A channel reads or writes data to a byte buffer. Channels are generally thread safe. A channel may be specified
to be read only or both readable and writable.
- ReadableByteChannel - A ReadableByteChannel permits read operation on a buffer, allowing only one thread to read at a time.
- WritableByteChannel - A WritableByteChannel permits write operation on a buffer, allowing only one thread to read at a time.
- ByteChannel - A ByteChannel extends both ReadableByteChannel and WritableByteChannel and allows both read and write.
- SeekableByteChannel - A SeekableByteChannel extends ByteChannel and allows to maintain and modify the current position on the underlying entity to which it is connected. It has methods to get the size of the underlying entity
or truncate it to a given size, if permitted.
- GatheringByteChannel-A GatheringByteChannel is used to write data from multiple buffers into a single channel.
- ScatteringByteChannel-A ScatteringByteChannel is used to read data from a channel into multiple buffers.
A FileChannel is used to read and write data to a file. It implements seekableByteChannel, ScatteringByteChannel and GatheringByteChannel. It is possible to map a region of file directly into memory. Data can be transferred to another channel or from another
channel using the transferTo(..) and transferFrom(..) methods. These methods use the underlying optimization of the operating system. File locking can be applied to manage access between multiple processes. The methods are thread safe and a thread that wishes
to modify the position may be blocked until another thread is acting upon the file.
Data from a File Channel can be read into multiple buffers. This is known as a scatter read. Here's and example demonstrating a scatter read.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
Path
tempFile =
Files.createTempFile( "fileChannelExample" , "txt" , new
FileAttribute<?>[ 0 ]);
PrintWriter
writer = new
PrintWriter(tempFile.toFile());
writer.println( "First
Line" );
writer.println( "Second
Line" );
writer.println( "Third
Line" );
writer.close();
FileInputStream
in = new
FileInputStream(tempFile.toFile());
FileChannel
fileChannel = in.getChannel();
System.out.println(fileChannel.position());
ByteBuffer
buffer = ByteBuffer.allocate( 11 );
fileChannel.read(buffer);
System.out.println(fileChannel.position());
System.out.println(fileChannel.size());
ByteBuffer
buffer1 = ByteBuffer.allocate( 14 );
ByteBuffer
buffer2 = ByteBuffer.allocate( 12 );
fileChannel.read( new
ByteBuffer[] { buffer1, buffer2 });
buffer1.flip();
buffer2.flip();
System.out.print(Charset.defaultCharset().decode(buffer1));
System.out.print(Charset.defaultCharset().decode(buffer2));
fileChannel.close();
|
we can write the bytes back to the file using a scattering write. The filechannel was created on the inputstream so the channel is only readable but not writable. we create a filechannel from an output stream
1
2
3
4
5
6
7
8
9
10
11
|
FileOutputStream
out = new
FileOutputStream(tempFile.toFile());
FileChannel
fileOutputChannel = out.getChannel();
ByteBuffer
buffer3 = Charset.defaultCharset().encode(
CharBuffer.wrap( "Line1\n" .toCharArray()));
ByteBuffer
buffer4 = Charset.defaultCharset().encode(
CharBuffer.wrap( "Line2" .toCharArray()));
fileOutputChannel.write( new
ByteBuffer[] { buffer3, buffer4 });
fileOutputChannel.force( true );
System.out.println(fileOutputChannel.position());
|
Java NIO allows reading huge files directly from the memory. i.e. The file need not be loaded into the JVM. All reads and writes on the byte buffer happen direclty on the file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
import
java.io.File;
import
java.io.IOException;
import
java.io.RandomAccessFile;
import
java.nio.MappedByteBuffer;
import
java.nio.channels.FileChannel;
public
class
ReadingHugeFilesUsingMemoryMappedBuffer {
/**
*
use a MappedByteBuffer to wrap a huge file. Using a MappedByteBuffer does
*
not load the file in JVM but reads it directly off the file system
*
memory. The file can be opened in read, write or private mode.
*/
private
static
String hugeFile = "A
Huge File" ;
public
static
void
main(String[] args) throws
IOException {
File
file = new
File(hugeFile);
FileChannel
fileChannel = new
RandomAccessFile(file, "r" ).getChannel();
MappedByteBuffer
buffer = fileChannel.map(
FileChannel.MapMode.READ_ONLY, 0 ,
fileChannel.size());
System.out.println(buffer.isLoaded());
System.out.println(buffer.capacity());
}
}
|