[ hadoop ] LeaseExpiredException No lease on ... File does not exist

최근 운영하고 있는 하둡 클러스터의 노후 장비 교체건으로 데이터노드 한대를 제거 하는 작업이 진행되었다.

해당 로그는 클러스터에 붙어 hdfs을 쓰고 있던 외부 서버에서 발생한 로그이다.

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/example....log._COPYING_ (inode 428475201): File does not exist. Holder DFSClient_NONMAPREDUCE_-1052637306_1 does not have any open files.

로그가 발생한 이유는 해당 서버가 hdfs 해당 데이터노드 특정 파일에 write작업을 하고 있었는데 중간에 데이터노드가 내려가면서 write하고 있던 파일을 찾지 못해 발생한 것이다. 

HDFS에서 Lease는 다음과 같이 정의된다.

In HDFS these locks are called Leases. Leases are granted to a client which request to open a file for a write operation (e.g. create / append / truncate a file.) Every lease belongs to a single HDFS Client but could be for several HDFS files. Often enough a lease has several thousand files open for write by a single HDFS client. As the client opens and closes files, the appropriate lease must be identified and updated. The exact datastructures have been changed quite frequently over the years to provide better lookups, better reverse lookups, speed and space efficiency etc. However all this accounting obviously is done on the NameNode. This is in stark contrast to GFS, where a lease is tracked by the Namenode (master server in their parlance) and Datanodes (chunk servers in their parlance) (Section 3.1 in the Google File System paper). For HDFS this means the Namenode has a higher overhead of now maintaining these leases (something GFS expressly wanted to avoid). However this also allows HDFS to allow renames of files being written (which in my experience is not too uncommon an operation.)

 

이 경우 외에도 스파크(Spark)나 hive의 병렬 작업시에도 작업이 꼬여 발생할 수 있다는 걸 검색을 하다가 알게되었다.

해당 내용은 아래의 블로그를 참고하길 바란다.

https://knight76.tistory.com/entry/hadoop-No-lease-on-File-does-not-exist

[hadoop] No lease on .. File does not exist.