Coin163

首页 > DiskLruCache源码详解

DiskLruCache源码详解

相关标签: cache 源码

2021腾讯云限时秒杀,爆款1核2G云服务器298元/3年!(领取2860元代金券),
地址https://cloud.tencent.com/act/cps/redirect?redirect=1062

2021阿里云最低价产品入口+领取代金券(老用户3折起),
入口地址https://www.aliyun.com/minisite/goods

相关推荐:Docker网络详解及pipework源码解读与实践

Docker作为目前最火的轻量级容器技术,有很多令人称道的功能,如Docker的镜像管理。然而,Docker同样有着很多不完善的地方,网络方面就是Docker比较薄弱的部分。因此,我们有必要深入了解Docker的网络知识,以满足更高的网络需求。本文首先介绍了Docker自身的

前言 DiskLruCache作为google认证通过的硬盘缓存实现方案,除了知道用法,还是有必要深究一下内部实现,有助于写出高质量的代码,以及清晰的框架实现思路。 源码解析 用过DiskLruCache的都知道,有journal这个文件的存在,内容一般如下: libcore.io.DiskLruCache111DIRTY e37775b7868532e0d2986b1ff384c078CLEAN e37775b7868532e0d2986b1ff384c078 152313 至于代表什么,等下再说 open() 观察源码发现,DiskLruCache的构造方法是private修饰,这也就是告诉我们,不能通过new DiskLruCache来获取实例,构造方法如下: private DiskLruCache(File directory, int appVersion, int valueCount, long maxSize) {

this.directory = directory;

this.appVersion = appVersion;

this.journalFile = new File(directory, JOURNAL_FILE);

this.journalFileTmp = new File(directory, JOURNAL_FILE_TEMP);

this.journalFileBackup = new File(directory, JOURNAL_FILE_BACKUP);

this.valueCount = valueCount;

this.maxSize = maxSize;} 但是提供了open()方法,供我们获取DiskLruCache的实例,open方法如下: /**

* Opens the cache in {@code directory}, creating a cache if none exists

* there.

*

* @param directory a writable directory

* @param valueCount the number of values per cache entry. Must be positive.

* @param maxSize the maximum number of bytes this cache should use to store

* @throws IOException if reading or writing the cache directory fails

*/

public static DiskLruCache open(File directory, int appVersion, int valueCount, long maxSize)

throws IOException {

if (maxSize <= 0) {

throw new IllegalArgumentException("maxSize <= 0");

}

if (valueCount <= 0) {

throw new IllegalArgumentException("valueCount <= 0");

}

// If a bkp file exists, use it instead.

//看备份文件是否存在

File backupFile = new File(directory, JOURNAL_FILE_BACKUP);

//如果备份文件存在,并且日志文件也存在,就把备份文件删除

//如果备份文件存在,日志文件不存在,就把备份文件重命名为日志文件

if (backupFile.exists()) {

File journalFile = new File(directory, JOURNAL_FILE);

// If journal file also exists just delete backup file.

//

if (journalFile.exists()) {

backupFile.delete();

} else {

renameTo(backupFile, journalFile, false);

}

}

// Prefer to pick up where we left off.

//初始化DiskLruCache,包括,大小,版本,路径,key对应多少value

DiskLruCache cache = new DiskLruCache(directory, appVersion, valueCount, maxSize);

//如果日志文件存在,就开始赌文件信息,并返回

//主要就是构建entry列表

if (cache.journalFile.exists()) {

try {

cache.readJournal();

cache.processJournal();

return cache;

} catch (IOException journalIsCorrupt) {

System.out

.println("DiskLruCache "

+ directory

+ " is corrupt: "

+ journalIsCorrupt.getMessage()

+ ", removing");

cache.delete();

}

}

//不存在就新建一个

// Create a new empty cache.

directory.mkdirs();

cache = new DiskLruCache(directory, appVersion, valueCount, maxSize);

cache.rebuildJournal();

return cache;

} open函数不难理解,一是,如果日志文件存在,直接去构建entry列表;二是,如果不存在,就构建日志文件。 先来看第二条分支:构建文件: rebuildJournal()

//这个就是我们可以直接在disk里面看到的journal文件 主要就是对他的操作 private final File journalFile; //journal文件的temp 缓存文件,一般都是先构建这个缓存文件,等待构建完成以后将这个缓存文件重新命名为journal private final File journalFileTmp;/**

* Creates a new journal that omits redundant information. This replaces the

* current journal if it exists.

*/

private synchronized void rebuildJournal() throws IOException {

if (journalWriter != null) {

journalWriter.close();

}

//指向journalFileTmp这个日志文件的缓存

Writer writer = new BufferedWriter(

new OutputStreamWriter(new FileOutputStream(journalFileTmp), Util.US_ASCII));

try {

writer.write(MAGIC);

writer.write("\n");

writer.write(VERSION_1);

writer.write("\n");

writer.write(Integer.toString(appVersion));

writer.write("\n");

writer.write(Integer.toString(valueCount));

writer.write("\n");

writer.write("\n");

for (Entry entry : lruEntries.values()) {

if (entry.currentEditor != null) {

writer.write(DIRTY + ' ' + entry.key + '\n');

} else {

writer.write(CLEAN + ' ' + entry.key + entry.getLengths() + '\n');

}

}

} finally {

writer.close();

}

if (journalFile.exists()) {

renameTo(journalFile, journalFileBackup, true);

}

//所以这个地方 构建日志文件的流程主要就是先构建出日志文件的缓存文件,如果缓存构建成功 那就直接重命名这个缓存文件,这样做好处在哪里?

renameTo(journalFileTmp, journalFile, false);

journalFileBackup.delete();

//这里也是把写入日志文件的writer初始化

journalWriter = new BufferedWriter(

new OutputStreamWriter(new FileOutputStream(journalFile, true), Util.US_ASCII));

} 这里就分析完毕了,再来看当日志文件存在的时候,做了什么 readJournal() private void readJournal() throws IOException {StrictLineReader reader = new StrictLineReader(new FileInputStream(journalFile), Util.US_ASCII);try {//读日志文件的头信息

String magic = reader.readLine();

String version = reader.readLine();

String appVersionString = reader.readLine();

String valueCountString = reader.readLine();

String blank = reader.readLine();

if (!MAGIC.equals(magic)

|| !VERSION_1.equals(version)

|| !Integer.toString(appVersion).equals(appVersionString)

|| !Integer.toString(valueCount).equals(valueCountString)

|| !"".equals(blank)) {

throw new IOException("unexpected journal header: [" + magic + ", " + version + ", "

+ valueCountString + ", " + blank + "]");

}//这里开始,就开始读取日志信息

int lineCount = 0;

while (true) {

try {

//构建LruEntries entry列表

readJournalLine(reader.readLine());

lineCount++;

} catch (EOFException endOfJournal) {

break;

}

}

redundantOpCount = lineCount - lruEntries.size();

// If we ended on a truncated line, rebuild the journal before appending to it.

if (reader.hasUnterminatedLine()) {

rebuildJournal();

} else {

//初始化写入文件的writer

journalWriter = new BufferedWriter(new OutputStreamWriter(

new FileOutputStream(journalFile, true), Util.US_ASCII));

}} finally {

Util.closeQuietly(reader);}} 然后看下这个函数里面的几个主要变量: //每个entry对应的缓存文件的格式 一般为1,也就是一个key,对应几个缓存,一般设为1,key-value一一对应的关系private final int valueCount;private long size = 0;//这个是专门用于写入日志文件的writerprivate Writer journalWriter;//这个集合应该不陌生了,private final LinkedHashMap<String, Entry> lruEntries =

new LinkedHashMap<String, Entry>(0, 0.75f, true);//这个值大于一定数目时 就会触发对journal文件的清理了private int redundantOpCount; 下面就看下entry这个实体类的内部结构

private final class Entry {

private final String key;

/**

* Lengths of this entry's files.

* 这个entry中 每个文件的长度,这个数组的长度为valueCount 一般都是1

*/

private final long[] lengths;

/**

* True if this entry has ever been published.

* 曾经被发布过 那他的值就是true

*/

private boolean readable;

/**

* The ongoing edit or null if this entry is not being edited.

* 这个entry对应的editor

*/

private Editor currentEditor;

@Override

public String toString() {

return "Entry{" +

"key='" + key + '\'' +

", lengths=" + Arrays.toString(lengths) +

", readable=" + readable +

", currentEditor=" + currentEditor +

", sequenceNumber=" + sequenceNumber +

'}';

}

/**

* The sequence number of the most recently committed edit to this entry.

* 最近编辑他的序列号

*/

private long sequenceNumber;

private Entry(String key) {

this.key = key;

this.lengths = new long[valueCount];

}

public String getLengths() throws IOException {

StringBuilder result = new StringBuilder();

for (long size : lengths) {

result.append(' ').append(size);

}

return result.toString();

}

/**

* Set lengths using decimal numbers like "10123".

*/

private void setLengths(String[] strings) throws IOException {

if (strings.length != valueCount) {

throw invalidLengths(strings);

}

try {

for (int i = 0; i < strings.length; i++) {

lengths[i] = Long.parseLong(strings[i]);

}

} catch (NumberFormatException e) {

throw invalidLengths(strings);

}

}

private IOException invalidLengths(String[] strings) throws IOException {

throw new IOException("unexpected journal line: " + java.util.Arrays.toString(strings));

}

//臨時文件創建成功了以後 就會重命名為正式文件了

public File getCleanFile(int i) {

Log.v("getCleanFile","getCleanFile path=="+new File(directory, key + "." + i).getAbsolutePath());

return new File(directory, key + "." + i);

}

//tmp开头的都是临时文件

public File getDirtyFile(int i) {

Log.v("getDirtyFile","getDirtyFile path=="+new File(directory, key + "." + i + ".tmp").getAbsolutePath());

return new File(directory, key + "." + i + ".tmp");

}} 好,到了这里,我们DiskLruCache的open函数的主要流程就基本走完了,那么就再走2个流程结束本篇的源码分析,当然了,一个是GET操作,一个是SAVE操作了。 我们先看get操作 get() /**

* Returns a snapshot of the entry named {@code key}, or null if it doesn't

* exist is not currently readable. If a value is returned, it is moved to

* the head of the LRU queue.

* 通过key获取对应的snapshot

*/

pub

相关推荐:史上最详细的LinkedHashMap详解--源码分析

史上最详细的LinkedHashMap详解–源码分析 ps.本文所有源码都是基于jdk1.6 LinkedHashMap数据结构 由下面代码可知,LinkedHashMap继承自HashMap。所以它保留了HashMap的数据结构,但是与之不同的是,它自己维护了一个双向循环链表,来保证LinkedHashMap的顺序

lic synchronized Snapshot get(String key) throws IOException {

checkNotClosed();

validateKey(key);

Entry entry = lruEntries.get(key);

if (entry == null) {

return null;

}

if (!entry.readable) {

return null;

}

// Open all streams eagerly to guarantee that we see a single published

// snapshot. If we opened streams lazily then the streams could come

// from different edits.

InputStream[] ins = new InputStream[valueCount];

try {

for (int i = 0; i < valueCount; i++) {

ins[i] = new FileInputStream(entry.getCleanFile(i));

}

} catch (FileNotFoundException e) {

// A file must have been deleted manually!

for (int i = 0; i < valueCount; i++) {

if (ins[i] != null) {

Util.closeQuietly(ins[i]);

} else {

break;

}

}

return null;

}

redundantOpCount++;

//在取得需要的文件以后 记得在日志文件里增加一条记录 并检查是否需要重新构建日志文件

journalWriter.append(READ + ' ' + key + '\n');

if (journalRebuildRequired()) {

executorService.submit(cleanupCallable);

}

return new Snapshot(key, entry.sequenceNumber, ins, entry.lengths);

} 再看一下,validateKey validateKey

private void validateKey(String key) {

Matcher matcher = LEGAL_KEY_PATTERN.matcher(key);

if (!matcher.matches()) {

throw new IllegalArgumentException("keys must match regex "

+ STRING_KEY_PATTERN + ": \"" + key + "\"");

}

} 这里是对存储entry的map的key做了正则验证,所以key一定要用md5加密,因为有些特殊字符验证不能通过。 然后看这句代码对应的: if (journalRebuildRequired()) {

executorService.submit(cleanupCallable);

} 对应的回调函数是:

/** This cache uses a single background thread to evict entries. */

final ThreadPoolExecutor executorService =

new ThreadPoolExecutor(0, 1, 60L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>());

private final Callable<Void> cleanupCallable = new Callable<Void>() {

public Void call() throws Exception {

synchronized (DiskLruCache.this) {

if (journalWriter == null) {

return null; // Closed.

}

trimToSize();

if (journalRebuildRequired()) {

rebuildJournal();

redundantOpCount = 0;

}

}

return null;

}

}; 其中的trimTOSize(): trimTOSize()

private void trimToSize() throws IOException {

while (size > maxSize) {

Map.Entry<String, Entry> toEvict = lruEntries.entrySet().iterator().next();

remove(toEvict.getKey());

}

} 就是检测总缓存是否超过了限制数量, 再来看journalRebuildRequired函数 journalRebuildRequired()

/**

* We only rebuild the journal when it will halve the size of the journal

* and eliminate at least 2000 ops.

*/

private boolean journalRebuildRequired() {

final int redundantOpCompactThreshold = 2000;

return redundantOpCount >= redundantOpCompactThreshold //

&& redundantOpCount >= lruEntries.size();

} 就是校验redundantOpCount是否超出了范围,如果是,就重构日志文件。 最后看get函数的返回值 new Snapshot()

/** A snapshot of the values for an entry. *///这个类持有该entry中每个文件的inputStream 通过这个inputStream 可以读取他的内容

public final class Snapshot implements Closeable {

private final String key;

private final long sequenceNumber;

private final InputStream[] ins;

private final long[] lengths;

private Snapshot(String key, long sequenceNumber, InputStream[] ins, long[] lengths) {

this.key = key;

this.sequenceNumber = sequenceNumber;

this.ins = ins;

this.lengths = lengths;

}

/**

* Returns an editor for this snapshot's entry, or null if either the

* entry has changed since this snapshot was created or if another edit

* is in progress.

*/

public Editor edit() throws IOException {

return DiskLruCache.this.edit(key, sequenceNumber);

}

/** Returns the unbuffered stream with the value for {@code index}. */

public InputStream getInputStream(int index) {

return ins[index];

}

/** Returns the string value for {@code index}. */

public String getString(int index) throws IOException {

return inputStreamToString(getInputStream(index));

}

/** Returns the byte length of the value for {@code index}. */

public long getLength(int index) {

return lengths[index];

}

public void close() {

for (InputStream in : ins) {

Util.closeQuietly(in);

}

}

} 到这里就明白了get最终返回的其实就是entry根据key 来取的snapshot对象,这个对象直接把inputStream暴露给外面。 最后再看看save的过程, public Editor edit(String key) throws IOException {

return edit(key, ANY_SEQUENCE_NUMBER);}//根据传进去的key 创建一个entry 并且将这个key加入到entry的那个map里 然后创建一个对应的editor//同时在日志文件里加入一条对该key的dirty记录private synchronized Editor edit(String key, long expectedSequenceNumber) throws IOException {

//因为这里涉及到写文件 所以要先校验一下写日志文件的writer 是否被正确的初始化

checkNotClosed();

//这个地方是校验 我们的key的,通常来说 假设我们要用这个缓存来存一张图片的话,我们的key 通常是用这个图片的

//网络地址 进行md5加密,而对这个key的格式在这里是有要求的 所以这一步就是验证key是否符合规范

validateKey(key);

Entry entry = lruEntries.get(key);

if (expectedSequenceNumber != ANY_SEQUENCE_NUMBER && (entry == null

|| entry.sequenceNumber != expectedSequenceNumber)) {

return null; // Snapshot is stale.

}

if (entry == null) {

entry = new Entry(key);

lruEntries.put(key, entry);

} else if (entry.currentEditor != null) {

return null; // Another edit is in progress.

}

Editor editor = new Editor(entry);

entry.currentEditor = editor;

// Flush the journal before creating files to prevent file leaks.

journalWriter.write(DIRTY + ' ' + key + '\n');

journalWriter.flush();

return editor;} 然后取得输出流 public OutputStream newOutputStream(int index) throws IOException {

if (index < 0 || index >= valueCount) {

throw new IllegalArgumentException("Expected index " + index + " to "

+ "be greater than 0 and less than the maximum value count "

+ "of " + valueCount);

}

synchronized (DiskLruCache.this) {

if (entry.currentEditor != this) {

throw new IllegalStateException();

}

if (!entry.readable) {

written[index] = true;

}

File dirtyFile = entry.getDirtyFile(index);

FileOutputStream outputStream;

try {

outputStream = new FileOutputStream(dirtyFile);

} catch (FileNotFoundException e) {

// Attempt to recreate the cache directory.

directory.mkdirs();

try {

outputStream = new FileOutputStream(dirtyFile);

} catch (FileNotFoundException e2) {

// We are unable to recover. Silently eat the writes.

return NULL_OUTPUT_STREAM;

}

}

return new FaultHidingOutputStream(outputStream);

}

} 注意这个index 其实一般传0 就可以了,DiskLruCache 认为 一个key 下面可以对应多个文件,这些文件 用一个数组来存储,所以正常情况下 我们都是 一个key 对应一个缓存文件 所以传0

//tmp开头的都是临时文件

public File getDirtyFile(int i) {

return new File(directory, key + "." + i + ".tmp");

} 然后你这边就能看到,这个输出流,实际上是tmp 也就是缓存文件的 .tmp 也就是缓存文件的 缓存文件 输出流。 这个流 我们写完毕以后 就要commit public void commit() throws IOException {

if (hasErrors) {

completeEdit(this, false);

remove(entry.key); // The previous entry is stale.

} else {

completeEdit(this, true);

}

committed = true;

}/这个就是根据缓存文件的大小 更新disklrucache的总大小 然后再日志文件里对该key加入clean的log//最后判断是否超过最大的maxSize 以便对缓存进行清理private synchronized void completeEdit(Editor editor, boolean success) throws IOException {

Entry entry = editor.entry;

if (entry.currentEditor != editor) {

throw new IllegalStateException();

}

// If this edit is creating the entry for the first time, every index must have a value.

if (success && !entry.readable) {

for (int i = 0; i < valueCount; i++) {

if (!editor.written[i]) {

editor.abort();

throw new IllegalStateException("Newly created entry didn't create value for index " + i);

}

if (!entry.getDirtyFile(i).exists()) {

editor.abort();

return;

}

}

}

for (int i = 0; i < valueCount; i++) {

File dirty = entry.getDirtyFile(i);

if (success) {

if (dirty.exists()) {

File clean = entry.getCleanFile(i);

dirty.renameTo(clean);

long oldLength = entry.lengths[i];

long newLength = clean.length();

entry.lengths[i] = newLength;

size = size - oldLength + newLength;

}

} else {

deleteIfExists(dirty);

}

}

redundantOpCount++;

entry.currentEditor = null;

if (entry.readable | success) {

entry.readable = true;

journalWriter.write(CLEAN + ' ' + entry.key + entry.getLengths() + '\n');

if (success) {

entry.sequenceNumber = nextSequenceNumber++;

}

} else {

lruEntries.remove(entry.key);

journalWriter.write(REMOVE + ' ' + entry.key + '\n');

}

journalWriter.flush();

if (size > maxSize || journalRebuildRequired()) {

executorService.submit(cleanupCallable);

}} 大家看那个32-40行,就是你commit以后 就会把tmp文件转正 ,重命名为 真正的缓存文件了。 这个里面的流程和日志文件的rebuild 是差不多的,都是为了防止写文件的出问题。所以做了这样的冗余处理。 特别感谢: 1.http://www.cnblogs.com/punkisnotdead/p/4815674.html

原文

前言 DiskLruCache作为google认证通过的硬盘缓存实现方案,除了知道用法,还是有必要深究一下内部实现,有助于写出高质量的代码,以及清晰的框架实现思路。 源码解析 用过DiskLruCache的都知道

------分隔线----------------------------