Tails

#1 Updated by emmapeel 2018-01-17 15:28:12

• blocks Feature #15079: Integrate the platform with our Git and ikiwiki infrastructure added

#3 Updated by emmapeel 2018-02-01 13:23:14

• Description updated

intrigeri wrote:
> Looks like weblate should detect that itself (probably based on Git’s own detection of it).

It detects it for the language files that are in master, but the problem are the ones only in weblate.

I have corrected the description to make it more clear.

#5 Updated by intrigeri 2018-02-13 19:04:50

• Deliverable for set to Sponsor_L

When doing Feature #15192 we noticed two more classes of related problems in the branch emmapeel submitted:

• 2 pages that were removed from our website still had .es.po files
• 50 pages that are not translatable anymore since 6-24 months still had .es.po files, which broke the News section on our English homepage

I don’t know if you want to track these two problems on this ticket or elsewhere, but IMO this is a blocker for calling the Weblate setup ready for production use.

#7 Updated by Anonymous 2018-02-14 09:56:42

• Assignee set to emmapeel
• QA Check set to Info Needed

@emmapeel: See my previous note. If you don’t want to handle it, you can reassign this to me so that we don’t lose track of this problem.

#10 Updated by emmapeel 2018-02-22 15:11:14

• Assignee deleted (emmapeel)

#11 Updated by Anonymous 2018-03-01 09:09:59

• Parent task set to Feature #10034

#12 Updated by Anonymous 2018-03-01 09:28:25

• related to Bug #15356: Automate adding new components to Weblate added

#13 Updated by Anonymous 2018-03-01 09:32:33

• Subject changed from weblate components need to be moved when moving files in git. to Weblate components need to be moved when files have been git mv'ed or deleted
• Description updated
• QA Check deleted (Info Needed)

#14 Updated by Anonymous 2018-03-02 15:34:14

• Subject changed from Weblate components need to be moved when files have been git mv'ed or deleted to Automatically update Weblate components on mdwn file add/modify/delete in tails.bo git
• Parent task changed from Feature #10034 to Feature #15082

This needs to happen between Weblate.git and Weblate (python got hook).

#15 Updated by Anonymous 2018-03-13 14:29:41

• related to Bug #15402: update hook for Main git when handling push from weblate added

#16 Updated by Anonymous 2018-05-29 13:12:32

• blocked by Bug #15624: Please install python-git, python3-git, python-pathlib on Weblate VM added

#17 Updated by Anonymous 2018-06-12 09:01:27

• Category set to Internationalization
• Assignee set to hefee

#18 Updated by hefee 2018-06-12 09:05:46

• Subject changed from Automatically update Weblate components on mdwn file add/modify/delete in tails.bo git to Automatically update Weblate components on mdwn file add/rename/delete in tails.bo git

#19 Updated by Anonymous 2018-06-12 14:12:18

• Status changed from Confirmed to In Progress

#20 Updated by hefee 2018-06-12 16:12:00

• % Done changed from 0 to 60

This script should do the handling of the components adding/deleting/renaming.
The rename part is not tested yet.

<code class="python">
import git                                                                                                                                                                                                                          
import logging                                                                                                                                                                                                                      
import os                                                                                                                                                                                                                           
import os.path                                                           
import pathlib
import sys  

def weblateComponentFile(filename, tree):                                               
    """is this file a weblate component"""                                                                      
    if not filename.startswith("wiki/src"):
        return False                                                     

    basename, *extensions = os.path.basename(filename).split(".")             
    if extensions and extensions[-1] == "po":                                  
        return False                 

    detectionFile = "{}.{}.po".format(os.path.splitext(filename)[0],LANG)                                                                                                                                                          
    try:
        tree/detectionFile
        return True
    except KeyError:
        return False
    return False


# Format logging    
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)  # Change to DEBUG if you wnat to see debug messages

formatter = logging.Formatter('%(levelname)s: %(message)s')
ch.setFormatter(formatter)                                             

logger = logging.getLogger('')
logger.setLevel(logging.DEBUG)                                   
logger.addHandler(ch)                                    

# DJANGO/WEBLATE setup to be able to change stuff within weblate
sys.path.insert(0,"/usr/local/share/weblate")          
os.environ["DJANGO_SETTINGS_MODULE"] = "weblate.settings"
os.environ["DJANGO_IS_MANAGEMENT_COMMAND"] = "1"    

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

from weblate.trans import models
from weblate.trans.management.commands import import_project                    

# weblate project name
PROJECT =  models.Project.objects.first().name

# repo path ( should be start with weblate:// )                            
REPO = models.SubProject.objects.first().repo

# Where is the git repository located?                 
PATH = pathlib.Path("/var/lib/weblate/repositories/vcs/tails/wikisrcindexpo")

# One langague that is activated and handled within the main git
LANG = "fr"

if not REPO.startswith("weblate://"):                             
    logger.error("We need a weblate:// url for searching for new components and not '{}'.".format(REPO))
    sys.exit(-1)

repo = git.Repo(str(PATH))                          

# fetch first, to see what we need to update
repo.remotes.origin.fetch()

# the object to orgin master branch
remote_master = repo.remotes.origin.refs.master

# create a diff                            
diff = repo.index.diff(remote_master.commit)

# perform pull                     
logger.info("HEAD before pull is at {}".format(repo.head.commit.hexsha))
repo.remotes.origin.pull()             

for deleted_file in diff.iter_change_type('D'):
    if not weblateComponentFile(deleted_file.a_path, tree=repo.tree()):
        logger.debug("deleted file (ignoring): {}".format(deleted_file.a_path))
        continue
    filemask = os.path.splitext(deleted_file.a_path)[0] + ".*.po"
    logger.info("deleted component: {}".format(filemask))
    subproject = models.SubProject.objects.filter(filemask=filemask).first()
    if subproject is None:
        logger.warning("\t... can't find component :(")
        continue
    if not subproject.repo.startswith("weblate://"):
        logger.warning("Do not delete {}, because it has no weblate url {}.".format(filemask, subproject.repo))
        continue
    subproject.delete()

# Handle added files
# as we rescan the whole dictonary anyways, we only need this once per dictonary

rescanDirs=set()
for added_file in diff.iter_change_type('A'):
    if not weblateComponentFile(added_file.a_path, tree=remote_master.commit.tree):
        logger.debug("added file (ignoring): {}".format(added_file.a_path))
        continue
    filemask = os.path.splitext(added_file.a_path)[0] + ".*.po"
    logger.info("added component: {}".format(filemask))
    fname = pathlib.Path(added_file.a_path)
    dirname = fname.parent
    rescanDirs.add(dirname)

for directory in rescanDirs:
    # Use import_project as used for inital creation of components
    # see /usr/local/sbin/weblate-setup-project.sh

    args=["--component-regex", str(directory/"(?P<name>.*)\.(?P<language>.+)\.po"),
        "--name-template", str(directory/"%s.*.po"),
        PROJECT,
        REPO,
        "master",
        str(directory/"**.*.po"),
        ]

    command = import_project.Command()
    parser = command.create_parser("",None)
    options = parser.parse_args(args=args)
    logger.info("searching for new components in {}".format(directory))
    command.handle(**vars(options))

# Handle renamed files (not tested yet)
for renamed_file in diff.iter_change_type('R'):
    if not weblateComponentFile(renamed_file.rename_to, tree=remote_master.commit.tree):
        logger.debug("renamed file (ignoring) {} -> {}".format(renamed_file.rename_from,renamed_file.rename_to))
        continue
    filemaskOld = os.path.splitext(renamed_file.rename_from)[0] + ".*.po"
    filemaskNew = os.path.splitext(renamed_file.rename_to)[0] + ".*.po"
    logger.info("renamed component {} -> {}".format(filemaskOld, filemaskNew))
    subproject = models.SubProject.objects.filter(filemask=filemaskOld).first()
    subproject.filemask = filemaskNew
    component.create_translations()  # Thats what loadpo is doing ...
</code>

#22 Updated by hefee 2018-06-14 12:28:03

• File update-weblate-components.py added

#24 Updated by hefee 2018-06-20 15:15:26

• related to Bug #15679: update puppet-tails to have scripts in weblate vm added

#26 Updated by emmapeel 2018-06-29 10:00:28

• related to Feature #15190: Cronjobs for translate.lizard are not running added

#27 Updated by hefee 2018-11-05 17:22:05

• related to Bug #15674: Change all Weblate components to use correct committer name and email added

#28 Updated by hefee 2018-12-14 03:10:27

• related to Feature #16225: merge hefee/dev/translation-server added

#29 Updated by hefee 2018-12-14 23:55:55

• Status changed from In Progress to Resolved

#30 Updated by hefee 2018-12-14 23:56:09

• % Done changed from 60 to 100

#32 Updated by Anonymous 2019-02-14 09:15:58

• Status changed from Resolved to In Progress

I’m reopening this ticket for the last question.

#33 Updated by hefee 2019-03-27 12:13:19

• Status changed from In Progress to Resolved

This should be solved with Bug #16490 got solved.

I know there are still some quirks around that make my scripts stop working. But those need to be addressed if they occur.

#34 Updated by intrigeri 2019-06-27 17:16:50

• Assignee deleted (hefee)