python代码词频分析(python实现词法分析器)
类别:脚本大全 浏览量:623
时间:2022-03-31 12:17:33 python代码词频分析
python实现词法分析器简单python词法分析器实现,供大家参考,具体内容如下
词法分析器状态转换图:
词法分析器总流程图:
预处理程序:
词法分析器:
词法分析器程序详细设计
详细代码实现:
|
#!/usr/bin/env python3.4 # coding=utf-8 import sys import string keywards = {} # 关键字部分 keywards[ 'false' ] = 101 keywards[ 'class' ] = 102 keywards[ 'finally' ] = 103 keywards[ 'is' ] = 104 keywards[ 'return' ] = 105 keywards[ 'none' ] = 106 keywards[ 'continue' ] = 107 keywards[ 'for' ] = 108 keywards[ 'lambda' ] = 109 keywards[ 'try' ] = 110 keywards[ 'true' ] = 111 keywards[ 'def' ] = 112 keywards[ 'from' ] = 113 keywards[ 'nonlocal' ] = 114 keywards[ 'while' ] = 115 keywards[ 'and' ] = 116 keywards[ 'del' ] = 117 keywards[ 'global' ] = 118 keywards[ 'not' ] = 119 keywards[ 'with' ] = 120 keywards[ 'as' ] = 121 keywards[ 'elif' ] = 122 keywards[ 'if' ] = 123 keywards[ 'or' ] = 124 keywards[ 'yield' ] = 125 keywards[ 'assert' ] = 126 keywards[ 'else' ] = 127 keywards[ 'import' ] = 128 keywards[ 'pass' ] = 129 keywards[ 'break' ] = 130 keywards[ 'except' ] = 131 keywards[ 'in' ] = 132 keywards[ 'raise' ] = 133 # 符号 keywards[ '+' ] = 201 keywards[ '-' ] = 202 keywards[ '*' ] = 203 keywards[ '/' ] = 204 keywards[ '=' ] = 205 keywards[ ':' ] = 206 keywards[ '<' ] = 207 keywards[ '>' ] = 208 keywards[ '%' ] = 209 keywards[ '&' ] = 210 keywards[ '!' ] = 211 keywards[ '(' ] = 212 keywards[ ')' ] = 213 keywards[ '[' ] = 214 keywards[ ']' ] = 215 keywards[ '{' ] = 216 keywards[ '}' ] = 217 keywards[ '#' ] = 218 keywards[ '|' ] = 219 keywards[ ',' ] = 220 # 变量 # keywards['var'] = 301 # 常量 # keywards['const'] = 401 # error # keywards['const'] = 501 signlist = {} # 预处理函数,将文件中的空格,换行等无关字符处理掉 def pretreatment(file_name): try : fp_read = open (file_name, 'r' ) fp_write = open ( 'file.tmp' , 'w' ) sign = 0 while true: read = fp_read.readline() if not read: break length = len (read) i = - 1 while i < length - 1 : i + = 1 if sign = = 0 : if read[i] = = ' ' : continue if read[i] = = '#' : break elif read[i] = = ' ' : if sign = = 1 : continue else : sign = 1 fp_write.write( ' ' ) elif read[i] = = '\t' : if sign = = 1 : continue else : sign = 1 fp_write.write( ' ' ) elif read[i] = = '\n' : if sign = = 1 : continue else : fp_write.write( ' ' ) sign = 1 elif read[i] = = '"': fp_write.write(read[i]) i += 1 while i < length and read[i] != '"' : fp_write.write(read[i]) i + = 1 if i > = length: break fp_write.write(read[i]) elif read[i] = = "'": fp_write.write(read[i]) i += 1 while i < length and read[i] != "'" : fp_write.write(read[i]) i + = 1 if i > = length: break fp_write.write(read[i]) else : sign = 3 fp_write.write(read[i]) except exception: print (file_name, ': this filename not found!' ) def save(string): if string in keywards.keys(): if string not in signlist.keys(): signlist[string] = keywards[string] else : try : float (string) save_const(string) except valueerror: save_var(string) def save_var(string): if string not in signlist.keys(): if len (string.strip()) < 1 : pass else : if is_signal(string) = = 1 : signlist[string] = 301 else : signlist[string] = 501 def save_const(string): if string not in signlist.keys(): signlist[string] = 401 def save_error(string): if string not in signlist.keys(): signlist[string] = 501 def is_signal(s): if s[ 0 ] = = '_' or s[ 0 ] in string.ascii_letters: for i in s: if i in string.ascii_letters or i = = '_' or i in string.digits: pass else : return 0 return 1 else : return 0 def recognition(filename): try : fp_read = open (filename, 'r' ) string = "" sign = 0 while true: read = fp_read.read( 1 ) if not read: break if read = = ' ' : if len (string.strip()) < 1 : sign = 0 pass else : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" sign = 0 elif read = = '(' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( '(' ) elif read = = ')' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( ')' ) elif read = = '[' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( '[' ) elif read = = ']' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( ']' ) elif read = = '{' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( '{' ) elif read = = '}' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( '}' ) elif read = = '<' : save(string) string = "" save( '<' ) elif read = = '>' : save(string) string = "" save( '>' ) elif read = = ',' : save(string) string = "" save( ',' ) elif read = = "'" : string + = read if sign = = 1 : sign = 0 save_const(string) string = "" else : if sign ! = 2 : sign = 1 elif read = = '"' : string + = read if sign = = 2 : sign = 0 save_const(string) string = "" else : if sign ! = 1 : sign = 2 elif read = = ':' : if sign = = 1 or sign = = 2 : string + = read else : save(string) string = "" save( ':' ) elif read = = '+' : save(string) string = "" save( '+' ) elif read = = '=' : save(string) string = "" save( '=' ) else : string + = read except exception as e: print (e) def main(): if len (sys.argv) < 2 : print ( "please input filename" ) else : pretreatment(sys.argv[ 1 ]) recognition( 'file.tmp' ) for i in signlist.keys(): print ( "(" , signlist[i], "," , i, ")" ) if __name__ = = '__main__' : main() |
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持开心学习网。
原文链接:https://blog.csdn.net/IT_DREAM_ER/article/details/53485583
您可能感兴趣
- python装饰器初学者教程(Python3.5装饰器原理及应用实例详解)
- python入门知识点总结(深入解析Python小白学习操作列表)
- pythonyield使用场景(Yii框架核心组件类实例详解)
- 怎么用python分析足球(使用Python进行体育竞技分析预测球队成绩)
- python mongodb 基本操作(Python使用pymongo库操作MongoDB数据库的方法实例)
- python数据结构全攻略(详解python数据结构和算法)
- 如何用python识别微信内容(Python 实现微信防撤回功能)
- python端口扫描脚本测试(Python实现的IP端口扫描工具类示例)
- python测试websocket接口(Python如何爬取实时变化的WebSocket数据的方法)
- python中怎么查看函数的用法(Python中函数的基本定义与调用及内置函数详解)
- python 3.10 循环语法(Python3.4学习笔记之常用操作符,条件分支和循环用法示例)
- python写一个二叉树(Python二叉搜索树与双向链表转换算法示例)
- python怎么用代码写出心形(六行python代码的爱心曲线详解)
- python读取文件的方法和区别(浅谈PYTHON 关于文件的操作)
- python中的多线程详解(python多线程抽象编程模型详解)
- python怎样看字符unicode编码(Python3中编码与解码之Unicode与bytes的讲解)
- 三兄妹感情再遇波折,人设接连崩塌 《以家人之名》剧情猜不透(三兄妹感情再遇波折)
- 《小敏家》金波想要复婚 这只是他圈套的第1步,更可恶的在后面(小敏家金波想要复婚)
- 小敏家 剧情离谱一锅乱炖,但他们俩绝对是这部剧的一大 亮点(剧情离谱一锅乱炖)
- 《倚天屠龙记》再遭翻拍,关晓彤主演赵敏,蒋劲夫演张无忌,你怎么看(倚天屠龙记再遭翻拍)
- 吴启华与曾舜晞两代张无忌同框,戏里经典的他却没活出原著的潇洒(吴启华与曾舜晞两代张无忌同框)
- 经常发这三种 朋友圈 的人,要迅速屏蔽(经常发这三种朋友圈)
热门推荐
- 云数据库navicat连不上(Navicat Premium15连接云服务器中的数据库问题及遇到坑)
- php如何发邮件(PHP示例演示发送邮件给某个邮箱)
- nginxdjango部署(详解Django+uwsgi+Nginx上线最佳实战)
- laravel 后台管理框架(关于Laravel-admin的基础用法总结和自定义model详解)
- sql server 数值类型运算(SQL SERVER2012中新增函数之字符串函数CONCAT详解)
- 怎么进入云服务器(云服务器端口怎么打开?)
- pandas数据分组后的处理(详解pandas如何去掉、过滤数据集中的某些值或者某些行?)
- css3定义自定义字体(CSS3字体效果的设置方法小结)
- dedecms列表分页代码(Dedecms文章设置推荐后列表页标题自动加粗的解决方法)
- python 写入d盘文件(python文件写入write的操作)
排行榜
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9